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PRE  FACE 

The  U.S.  Army  Computer  Systems  Command  (USACSC)  has  been  addressing  Software 
Life  Cycle  Management  (SLCM)  problems  emanating  from  current  systems.  These 
problems  are  likely  to  become  even  more  significant  because  of  the  requirements  for 
increasingly  complex  and  costly  software  systems.  Recently  it  has  recognized  that 
theoretical  and  experimental  work  in  the  area  of  software  phenomenology  is  strongly 
related  to  SLCM.  Since  many  separate  researchers  have  been  employing  the  global, 
phenomenological  approach,  it  seemed  appropriate  and  timely  to  bring  them  together 
to  present  their  ideas,  to  engage  in  discussions,  to  establish  any  relationships  that  may 
exist  between  various  concepts  and  techniques,  and  to  recommend  promising  solutions 
and  prospective  research  directions. 

To  achieve  this  forum,  USACSC,  through  its  Army  Institute  for  Research  in  Management 
Information  and  Computer  Science  (AIRMICS)  and  the  Integrated  Software  Research 
and  Development  (ISRAD)  Program,  decided  to  sponsor  a workshop  addressing  the  many 
facets  of  phenomenology  and  of  software  life  cycle  management.  The  workshop  was  to 
focus  upon  practical  solutions  to  current  SLCM  problems,  emphasizing  short-  and  long- 
range  approaches  supported  by  quantitative  studies.  Its  format  was  designed  to 
encourage  discussions  of  current  problems  and  to  elicit  new  perspectives  on  potential 
solutions  through  the  interaction  of  researchers  with  varying  views.  The  desirability 
of  achieving  practical  results  implied  strong  restrictions  on  the  number  of  attendees, 
unavoidably  excluding  from  attendance  a number  of  researchers  whose  work  would  have 
contributed  to  the  workshop's  overall  quality.  It  is  hoped  that  their  participation  will 
be  possible  at  subsequent  workshops.  Approximately  30  international  experts  in  four 
* general  areas  were  invited  to  present  and  discuss  papers  and  to  recommend 

prospective  research  directions. 

! 

This  volume  presents  the  submissions  to  the  Workshop  provided  by  the  individual  authors. 
^ No  attempt  has  been  made  by  the  editors  to  unify  the  styles  or  to  referee  the  papers 

formally,  but  they  have  been  ordered  in  what  appears  to  be  a fairly  natural  sequence. 

I j The  volume  also  includes  an  executive  summary  with  recommendations,  a summary  of 
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EXECUTIVE  SUMMARY 


The  Software  Life  Cycle  Management  Workshop  was  conceived  and  directed  by  Colonel 
Lawrence  Putnam  and  Professor  Meir  Lehman.  Through  their  efforts,  33  speakers  and 
17  observers  gathered  at  the  Airlie  House  in  Warrenton,  Virginia  for  a 2-day  workshop 
addressing  quantitative  aspects  of  the  software  development  process.  The  workshop 
was  organized  into  four  parallel  groups  addressing  problems  in  the  areas  of  Measure- 
ment of  Quality  and  Reliability.  Management  Control  and  Resource  Allocation.  I^henon- 

I 

enology  of  acfhvare  Development  and  Maintenance,  and  Human  Factors.  Individaal  and 
Group  Product! vltv  Measures.  The  workshop  opened  with  an  introduction  by  Colonel 
Putnam,  who  stated  the  workshop  objectives:  (1)  to  identify  problems;  (2)  to  share  ideas 
about  promising  solutions  to  the  problems;  and  (3)  to  recommend  prospective  research 
directions  leading  to  solutions.  The  technical  introduction  by  Professor  Lehman  that 
followed  set  the  scene  for  the  two  days  of  discussions,  and  its  text  is  included  in  the 
proceedings.  At  the  end  of  two  days  of  parallel  and  general  sessions,  which  included 
an  after-dinner  address  by  Dr.  John  Staudhammer  entitled  "On  Software  Engineering," 
a plenary  session  convened  to  summarize  the  results  of  each  of  the  four  sessions  and 
to  review  the  degree  of  success  in  achieving  the  workshop  objectives.  Recommendations 
made  at  the  plenary  session  are  listed  below  and  are  explained  in  greater  detail  in  the 
plenary  session  summary. 

Recommendations  and  Prospective  Research  Directions 
Immediate  emphasis  should  be  given  in  the  following  areas: 

• Standards  for  system  design,  documentation,  process  measure  definition, 
measurement,  and  other  software  development  processes 

• Determining  and  defining  measurable  attributes  of  software  quality  and  process 
productivity 

• Data  Collection  and  analysis  as  a continuing  process 

• Understanding  discontinuities  resulting  from  project  size  and  complexity 

• Flexibility  as  a system  design  attribute 
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• Design  ^nd  construction  of  software  modules  for  re-usabilit>' 

• Refining  configuration  management  to  provide  a simple,  direct  procedure  for 
maintaining  and  controlling  software  system  structure 

• Verification  and  extension  of  techniques  such  as  those  arising  from  the 
Rayleigh  curve  cost  model  and  from  the  results  of  the  Evolution  Dvnamics 


Studies  to  establish  their  usefulness  over  a greater  range  of  projects 

• Implementation  of  life  cycle  models  in  the  management  control  process 

• Impact  of  environment,  change  and  module  integration  upon  software 
reliability 

• Testing  and  using  existing  software  reliability  models. 

Future  Research  Recommendations 


Future  software  research  and  development  should  be  directed  to  the  following  areas, 
each  of  which  promises  added  understanding  and  control  over  the  software  development 
process: 


System  metrics 

System  Evolution  Dynamics  and  its  interpretation 

Value  assessment  of  proposed  methodologies 

Development  of  tools  to  fill  defined  needs 

Maintainability  (definitions,  models,  techniques  and  practices) 

System  reliability  (hardware,  software,  human) 

The  implementation  of  change 

Software  Science  and  Physics  (in  the  sense  of  Halstead  and  Kolence). 


Future  Activity 

At  the  conclusion  of  the  plenary  session,  it  was  recommended  that  a second  workshop 
be  held  to  continue  the  discussions  and  to  explore  more  deeply  the  phenomenology  of 
software  development. 
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SLCM  WORKSHOP-TECHNICAL  INTRODUCTION 
M.  M.  LEHMAN 


In  presenting  the  technical  introduction  this  Workshop  on  Software  Life  Cycle  Manage- 
ment I should  first  try  to  explain  why  you  have  been  invited  and  what  we  and  the  spon- 
sors, AIRMICS  and  ISRAD,  hope  to  achieve. 

The  meeting  was  conceived  during  a discussion  between  Colonel  Putnam  and  myself 
at  the  2nd  International  Conference  on  Software  Engineering  in  San  Francisco  in 
October  1976.  We  each  realized  that,  despite  a difference  of  focus,  his  work  on  life 
cycle  cost  distribution  was  strongly  related  to  work  by  Les  Belady,  Francis  Parr  and 
myself  in  Software  Evolution  Dynamics.  The  connection  related  both  to  the  methodology 
used  and  to  global  conclusions  reached.  This  same  observation  was  made  also  by  an 
independent  observer  who  wrote  in  a review  of  the  conference  in  the  December  1976 
issue  of  Computer,  "Interpreting  this  result  together  with  hfs  earlier  project  estima- 
tion model,  Putnam  derived  virtually  the  same  management  implications  as  those 
presented  by  Lehman:  Large  system  development  is  inherently  stable  and  will  be 
driven  to  the  natural  parameters  of  the  process;  . . . the  only  real  management 
options  to  influence  the  systems  are  Go,  No-Go,  Limit,  and  Freeze  Decision." 

In  our  brief  talk,  Larry  and  I observed  that  the  technical  basis  of  our  common  approach 
was  pheno meno logy , the  observation  of  the  real  world  as  it  is.  Measurement  of  its 
behavior,  development  of  best-fit  models  that  describe  but  do  not  explain  thai  behavior, 
interpretation  of  the  models,  and  the  formulation  and  experimental  confirmation, 
modification  or  rejection  of  theories  will  provide  the  basis  of  a software  science  and 
engineering  discipline.  The  whole,  of  course,  is  a highly  iterative  procedure.  I shall 
return  to  this  in  a moment,  but  first,  let  me  continue  my  historical  review.  We  also 
realized  that  there  were  other  workers  in  Computer  Science  and  Software  Engineering 
adopting  the  same,  or  a related  approach.  Brooks  in  his  Mythical  Man-Month  adopts  a 
global  view.  Kolence  and  Halstead  in  what  they  have  termed  "software  physics"  and 
"software  science"  respectively,  adopt  the  measurement,  modeling  and  interpretation 


approach,  as  do  people  like  Musa  and  Shooman  in  the  reliability  field.  Many  more 


examples  could  be  cited.  We  tried  to  invite  such  globalists  to  this  meeting,  but  all 
could  not  attend.  My  apologies  to  those  whom  we  were  unable  to  locate. 


So  much  as  to  the  background.  What  is  to  be  the  subject  matter  of  our  discussions  ? As 
1 have  indicated,  in  the  scientific  sense  our  universe  of  discourse  is  the  phenomenology 
of  software  development  and  maintenance.  More  pragmatically,  we  wish  to  direct 
attention  to  the  meaning  of,  the  need  for,  and  the  methodologies  of  Life  Cycle  Manage- 
ment of  the  software  development  and  maintenance  processes. 

When  questions  of  cost,  throughput,  reliability  (or  better,  uncertainty)  and  malleability 
are  addressed,  one  may  not  focus  exclusively  on  one  activity,  on  the  execution  time  of 
one  program  or  on  the  adaptability  of  the  system  at  one  moment  in  time.  The  life 
cycle  of  an  application,  of  a program  through  its  various  releases,  even  of  the  process 
whereby  an  individual  program  is  developed  and  maintained,  is  multi-faceted.  There 
is  continuous  potential  for  tradeoff,  of  balance  between  local  and  global  considerations, 
short-term  and  long-term  implications.  All  program  management  occurs  in  an  environ- 
ment of  continuous  evolution  of  the  application,  the  technologies  and  the  system.  * 

Evolution  is  an  intrinsic  property  of  computer  usage. 

It  follows  that  computer  science  activity  must  at  least  in  part  be  based  on  observation, 
measurement  and  analysis  of  the  total  lifetime  of  systems  and  over  systems.  That  is, 
de  facto,  there  exists  an  approach  to  computer  science  and  software  engineering  based 
on  the  concept  and  methodology  of  phenomenology. 

Let  me  digress  for  a moment.  Any  engineering  discipline  is  based  on  a combination  of 
pragmatics,  natural  science  and  mathematics.  In  figure  1 I have  tried  to  summarize 
my  view  of  some  major  relationships  between  the  real  world,  mathematics,  science  and 
engineering  practice.  In  particular,  I wish  to  draw  attention  to  the  major  role  played 
by  measurement  and  phenomenology,  particularly  during  the  formative  stages  of  a 
new  discipline.  I can  do  no  better  here  than  give  you,  in  figure  2,  two  quotations 
that  eloquently  express  and  summarize  what  I have  been  trying  to  say. 
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NATURE  OF  ENGINEERING  SCIENCES 
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- LARGELY  DEDUCTIVE 

- BOTTOM  UP 

- SYNTHESIS 

Figure  1 
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PHENOMENOLOGY 


"THE  WHOLE  TREND  OF  MODERN  SCIENTIFIC  VIEWS  IS  TO 

BREAK  DOWN  THE  SEP.ARATE  CATEGORIES TO  SUBSTITUTE  A 

COMMON  BACKGROUND  OF  ALL  EXPERIENCE OUR  SCIENTIFIC 

INFORMATION  IS  SUMMED  UP  IN  MEASLUES WE  FEEL  IT  NECESSARY 

TO  CONCEDE  SOME  BACKGROUND  TO  THE  MEASURES  - AN  EXTERNAL 
WORLD;  BUT  THE  ATTRIBUTES  OF  THIS  WORLD.  EXCEPT  IN  SO  FAR  AS 
THEY  .\RE  REFLECTED  IN  THE  MEASURES.  .\RE  OUTSIDE  SCIENTIFIC 
SCRUTINY. " 

Eddington  - Nature  of  the  Physical  World 


MEASLUEMENT 

"I  OFTEN  SAY  THAT  WHEN  YOU  C.\N  MEASURE  WHAT  YOU  ARE 
SPEAKING  ABOUT.  .\ND  EXPRESS  IT  IN  NUMBERS.  YOU  KNOW  SOMETHING 
ABOUT  IT;  BUT  WHEN  YOU  CANNOT  EXPRESS  IT  IN  NUMBERS,  YOUR  KNOWL- 
EDGE IS  OF  A MEAGER  AND  UNSATISF.ACTORY  KIND;  IT  MAY  BE  THE  BEGIN- 
NING OF  KNOWLEDGE.  BUT  YOU  HAVE  SC.ARCELY,  IN  YOUR  THOUGHTS, 
.\DV.\NCED  TO  THE  STAGE  OF  SCIENCE.  WHATEVER  THE  MATTER  MAY  BE." 


Lord  Kelvin 


^ - - --  - 


The  term  Computer  Science  has  been  around  since  the  early  sixities.  "Software  Engi- 
neering" was,  I believe,  first  coined  in  connection  with  the  1968  NATO  conference  by  that 
name.  We  may  now  legitimately  ask — indeed,  may  have  been  asking  for  a number  of 
years — whether  these  terms  are  indeed  justified  and  what  the  science  and  the  engineering 
disciplines  really  are.  The  pressure  for  measurement  has  been  around  in  our  commun- 
ity for  a number  of  years,  particularly  but  not  exclusively  in  the  area  of  performance 
evaluation.  But  the  phenomenological  investigations  have  been  restricted  to  individuals 
working  largely  in  isolation  or  in  very  small  groups. 

Our  intentions  were  to  identify  as  many  of  these  individuals  as  we  could  and  to  bring 
them  together  to  learn  from  each  other;  to  challenge  one  another;  to  share  experience, 
observation  and  conclusions;  to  identify  and  develop  intersections  and  common  view- 
points; to  exchange  tools  and  techniques;  and  hopefully,  to  e.xtract  some  basic  principles 
and  generalizations. 

These  are  worthy  and  ambitious  aims  in  themselves.  But  under  encouragement  from 
our  sponsor,  we  hope  to  go  one  step  further.  The  meeting  should  seek  to  identify  prin- 
ciples, methodologies,  techniques  and  tools  that  are  ready  for  exploitation.  Additionally, 
we  would  hope  to  identify  potentially  fruitful  research  directions  in  software  engineering, 
programming  methodology,  software  management  and  in  applied  computer  science.  That 
is,  we  expect  both  to  suggest  areas  and  directions  of  investigation  and  research  that  will 
advance  the  development  of  computer  science  and  software  engineering  as  intellectual 
disciplines  and  also  to  develop  concepts,  methodologies,  techniques  and  tools  that  are 
ready  for  applications;  that  will  prove  of  significant  value  to  the  practitioners. 

I have  already  taken  up  far  too  much  of  our  very  restricted  time.  Let  us  now  get  down 
to  our  business  following  the  format  that  Colonel  Putnam  has  outlined,  but  with  willing- 
ness to  change,  to  apply  management  feedback  and  control  so  as  to  maximize  our  col- 
lective intellectual  gain  and  our  real  output. 
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PLENARY  SESSION  SUMMARY 

CHAIRMAN:  Meir  M.  Lehman 

PANELISTS:  Robert  McHenry,  Kenneth  Kolence,  Les  Belady,  Maurice  Halstead 

Manny  Lehman  pointed  to  the  successful  achievement  of  goals  for  the  1977  SLCM 
Workshop  and  the  unanimous  desire  of  participants  that  a second  workship  be  convened 
early  in  1978, 

Each  chairman  was  then  asked  to  review  activities  during  the  technical  sessions. 

Phenomenology  of  Software  Development  and  Maintenance 
Chairman:  Les  Belady 

Les  Belady  presented  his  session's  report,  which  was  organized  in  top-down  fashion 
(total  system  view,  system  structure/process,  and  methodologies),  followed  by 
conclusions  and  recommendations. 

Each  of  the  issues  addressed  within  the  session  had  a common  objective  to  improve 
overall  software  management  capabilities.  Specifically,  the  discussants  addressed 
three  key  management  metrics  — predictability,  controllability,  and  cost-effectiveness. 
It  was  agreed  that  a phenomenology'  of  software  development  was  needed  and  that  its 
existence  was  evidenced  throughout  the  workshop  discussions. 

Total  system  view  of  software  - A total  system  view  of  software  was  stressed,  and 
observations  of  software  development  based  upon  the  following  three  perspectives 
were  discussed: 

• the  stages  in  user  growth 

• the  total  life  cycle  of  a software  development  project 

• the  single  release  process 
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It  was  noteworthy  that  each  perspective  was  supported  by  observed  data  on  actual  sys- 
tem developments. 

System  structure/process  - To  achieve  the  controlled  development  of  software  systems, 
the  properties  unique  to  the  system  structure  must  be  identified,  the  totality  of  inter- 
relationships (hardware,  software,  human)  must  be  thoroughly  understood,  and  full  use 
of  all  potential  feedback  processes  must  be  made.  Knowledge  in  these  areas  is  necessary 
to  enable  the  system  content,  structure  and  development  process  to  change  as  the  sys- 
tem requirements  change. 

Methodology  - To  aid  in  the  software  development  process,  common  standards  must  be 
established  for  clear,  unequivocal  communications  links  between  the  various  groups 
involved. 

General  procedures  which  promote  greater  management  visibility  and  control  must  be 
established  throughout  the  software  development  process.  Such  overall  procedures  can 
be  supplemented  by  project  specific  procedures  as  required.  Tools  which  support  all 
phases  of  the  software  development  process  should  be  developed  and  made  easily 
accessible.  One  attempt  was  made  to  assess  the  value  of  a number  of  software  develop- 
ment tools  and  methodologies  based  upon  their  implementation  cost  and  value  to  the 
overall  development  process.  E.xplicit  tests  of  other  proposed  tools  should  be  conducted, 
so  that  their  value  to  the  life  cycle  process  can  be  determined. 

Conclusions  and  recommendations  generated  during  the  session  are  listed  below: 

Conclusions  - 

• The  software  development  process  can  be  described  by  a (partially  known) 
measurable,  generic  set  of  phenomena. 


• Each  project  requires  a management  system  specific  to  its  environment, 
which  is  based  upon  the  generic  phenomena  and  uses  locally  generated 
measure  values  for  planning  and  project  control. 


• Any  organization  can  adopt  and  continually  improve  uniform  procedures 
and  methodologies  which  fit  all  their  projects  and  systematically  apply 
the  generic  phenomena  within  the  management  system  selected. 


Recommendations  - The  recommendations  set  forth  by  the  session  are  divided  into  two 
sets  — those  pertinent  to  "Problem  Management"  and  those  pertinent  to  "Problem 

I .Avoidance. " 

I Problem  Management  - 

• Data  collection  and  analysis  should  be  a continuous  process  associated 
with  any  software  project  and  should  be  reported  to  the  software  develop- 
ment community  for  general  use. 

• Standards  for  system  design,  documentation,  measurement  and  other 

• processes  associated  with  software  development  should  be  promulgated. 

• Configuration  management  must  be  studied  and  refined  to  provide  a 
simple,  direct  procedure  to  maintain  and  control  software  system  plans. 

'1 

• Education  is  a necessary  ingredient  in  the  successful  software  develop- 

; ment  process. 
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Problem  Avoidance  - 

• Size/complexity  of  projects  have  been  identified  as  key  parameters  in 
determining  overall  management  procedures  necessary  for  successful 
project  control.  An  observable  discontinuity  in  these  two  parameters 
requires  greater  understanding.  Considerable  research  is  required  to 
clearly  establish  these  parameters,  their  range,  and  their  relationship 
to  the  overall  project. 

• Changeability  must  be  carefully  designed  into  systems  in  order  to  allow 
for  the  unavoidable  user  requirements  shifts  and  management-determined 
redirections  during  the  life  cycle  of  a software  system. 
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• Reusability  of  software  modules  should  become  an  established  practice. 
This  includes  the  development  of  software  libraries  and  a communications 
mechanism  to  assure  that  such  software  libraries  are  fully  e-xploited  by 
system  developers. 

Recommended  Research  - The  session  recommended  four  specific  areas  for  future 
software  research  and  development: 

• System  metrics 

• System  invariants  or  regularities 

• Value  assessment  of  proposed  methodologies 

• Development  of  tools  to  fill  defined  needs. 

Management  Control  and  Resource  Allocation 
Chairman:  Kenneth  W.  Kolence 

The  working  group  had  17  people  or  more  in  attendance  at  each  of  the  working  sessions, 
of  whom  7 were  "observers"  and  10  had  been  invited  to  present  papers.  Following  the 
normal  procedures,  each  of  the  10  (with  one  exception)  gave  a 10-15  minute  brief  over- 
view of  their  papers.  The  exception  was  the  session  chairman,  K.  W.  Kolence,  who 
felt  his  paper  was  not  worthwhile  presenting  due  to  both  a shortage  of  time  and  the  fact 
the  group  had  already  focused  on  the  project  life  cycle  curves  as  the  primary  topic  to 
be  considered.  A list  of  the  papers  and  presenters  at  the  first  two  sessions  is  included 
as  an  appendix  to  this  report. 

There  were  three  people  whose  papers  directly  addressed  the  use  of  Rayleigh  curves  as 
a model  of  the  manpower-time  relationships  in  design  and  related  project  efforts.  Of 
these,  the  papers  of  Larry  Putnam  and  Peter  Norden  were  heavily  oriented  toward 
explanation  of  observed  empirical  data  in  terms  of  the  Rayleigh  equations.  J.  Spruce 
Riordon's  paper  proposed  a process  control  model  of  the  equation  to  investigate  its 
dynamic  characteristics  and  represents  an  extension  of  the  work  of  Belady  and  Lehman. 
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In  addition  to  the  above-mentioned  papers,  those  of  Ian  Elliott,  Clair  Miller,  and  Jules 
Schwartz  also  addressed  the  basic  questions  related  to  the  use  cf  Rayleigh  curves  in 
practice.  In  particular,  the  constraints  on  available  manpower  and  management  alloca- 
tion practices  are  the  topics  of  the  latter  two  papers.  Elliott's  paper  concerns  the  impli- 
cations arising  from  the  overall  age  of  code  within  the  life  cycle  on  the  manning  levels 
within  a programming  organization.  All  three  of  these  papers  relate  directly  to  the 
Rayleigh  curve  papers  in  at  least  some  way,  so  the  majority  of  the  basic  group  of  10 
papers  were  relatable.  Each  of  the  other  four  papers  were  on  different  topics,  but 
again  generally  relatable  to  the  topic  of  Rayleigh  curves. 

Norden,  Putnam,  and  Riordon  were  requested  by  the  group  to  give  detailed  presentations 
of  their  papers,  which  they  did  in  the  order  given  above.  It  was  the  clear  consensus  of 
the  group  that  Norden's  and  Putnam's  presentations  were  convincing  evidence  that  Ray- 
leigh curves  were  empirically  shown  to  represent  the  manpower-time  relationships  in 
design  projects.  The  research  recommendations  of  the  group  were  therefore  based  on 
the  belief  that  a fundamentally  important  empirical  relationship  had  been  established  for 
the  software  life  cycle  concept.  However,  a variety  of  important  anomalies  and  discrep- 
ancies relative  to  the  real  world  were  commented  on,  and  need  to  be  taken  into  serious 
consideration  in  adapting  these  curves  for  general  software  engineering  use.  These  are 
discussed  below,  in  no  particular  order  of  importance.  Many  of  these  were  commented 
on  in  the  group  discussions,  but  some  are  the  result  of  subsequent  reading  and  consider- 
ations by  the  session  chairman.  Those  in  the  latter  category  are  clearly  identified. 

In  the  presentation  by  Norden,  the  Rayleigh  curves  were  empirically  and  logically  shown 

to  be  valid  only  for  each  activity  which  coherently  reflects  a process  of  making  a large 

set  of  decisions.  For  example,  a requirements  phase,  a design  phase,  a prototype 

development  phase  (for  equipment)  etc.  The  combination  of  phases  into  a single  overal. 

activity  did  not  form  a Rayleigh  curve,  nor  logically  could  it.  Putnam's  empirical  data 

for  major  software  development  projects  showed  the  overall  activity  was  reflected  by  a | 

Rayleigh  curve.  The  strong  implication  here  is  that  the  current  division  of  a software  | 
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life  cycle  into  individual  phases  is  not  being  implemented  in  practice  in  a way  which 
represents  separate  decision  making  activities.  That  is,  the  specification-design- 
implementation  techniques  in  use  do  not  reflect  actually  distinct  processes  of  decision 
making;  e.g. , the  coding  process  includes  elements  of  the  specification  process.  If 
this  is  so,  and  clearly  practice  would  support  this  thesis,  then  a rather  thorough  re- 
examination of  current  "software  design  methodologies"  is  obviously  needed  to  avoid 
"code-level  specification  decisions."  This  is  perhaps  the  most  fundamental  anomaly 
which  was  brought  to  the  group's  attention. 

.A,  second  set  of  problems  of  practical  importance  was  also  discussed.  The  Rayleigh 
curves  have  been  used  successfully  to  predict  the  total  manpower  needs  and  elapsed 
time  of  a decision-making  phase  (e.g. , a full  software  development  cycle).  The  two 
parameters  needed  are  1)  the  total  manpower  expenditure  K,  and  2)  the  curve  "shape 
parameter  a,  which  represents  the  peak  manpower  loading.  Given  these,  tracking  of 
actuals  to  planned  and  fitting  early  actual  data  to  the  Rayleigh  curve  equation  can  project 
actual  total  manpower,  peak  loadings,  and  total  elapsed  time  with  good  accuracy. 
.Alternately,  given  a project  with  a good  estimate  of  the  total  effort  required  and  the 
schedule  objectives  for  completion,  the  manpower  loading  necessary  can  be  determined. 
Further,  Putnam's  work  shows  that  certain  regions  of  the  manpower-time  planes  are 
"forbidden"  in  the  sense  that  actual  projects  cannot  reflect  these  combinations.  In 
other  words,  a thousand  man-month  effort  cannot  be  completed  in  one  month  by  assigning 
a thousand  people  to  it.  Putnam  has  interpreted  this  to  mean  that  systems  have  an 
inherent  "degree  of  difficulty"  which  severely  constrains  the  possible  manpower- 
completion  time  parameters  of  any  gfiven  software  project. 

The  problems  which  arise  from  these  considerations  are  both  practical  and  theoretical. 
On  the  practical  side,  use  of  these  curves  to  originally  project  the  time  required  to  com- 
plete an  effort  depends  critically  on  the  estimated  man-months.  Reasonably  accurate 
man-month  estimates  can  be  made  a priori  only  when  the  group  responsible  has  suf- 
ficient previous  experience  with  similar  projects.  New  types  of  efforts  by  an  organiza- 
tion (e.g. , an  interactive  online  system  as  opposed  to  batch  systems)  cannot  usually  be 
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estimated  with  any  reasonable  accuracy.  In  other  words,  if  you've  done  it  before,  you 
know  how  long  it  will  take.  If  \’OU  haven't,  you  don't.  In  addition,  there  are  clearly 
learning  curve  variations  in  the  effort  required. 
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On  the  theoretical  side,  the  use  of  the  quantity  K (total  man-months)  as  a variable  to 
represent  the  "comple.xity”  of  a design  effort  is  untenable  on  a variety  of  grounds.  Fund- 
amentally, two  things  are  being  represented  by  K:  1)  the  design  complexity,  i.  e. , the 
number  of,  and  perhaps  interactions  between,  the  design  choices,  and  2)  the  ability  of 
the  individuals  involved  to  recognize  the  need  for  these  choices  and  the  rate  of  correctly 
making  them.  The  variable  K combines  these,  whereas  for  predictive  purposes  one 
would  like  them  separated.  This  would  appear  (to  the  session  chairman)  to  be  a fruitful 
practical  research  area,  as  it  would  lead  to  a better  understanding  of  the  practical 
meaning  of  the  term  "design  complexity,"  and  a means  of  evaluating  the  relative  capa- 
bilities of  different  design  teams  to  successfull}’^  complete  an  effort.  It  may  also  provide  ■ 

a means  for  evaluating  the  effectiveness  of  different  "design  methodologies,"  as  was  ' 

noted  in  the  group  when  Norden  presented  some  comparative  data  using  different  metho- 
dologies. 

A second  theoretical  question  with  significant  practical  importance  is  the  understanding 
of  the  term  "similar  designs."  In  estimating  K,  this  is  an  important  consideration.  It 
likewise  is  of  importance  in  understanding  the  theoretical  term  "design  complexity." 

The  group  discussed  this  question  briefly,  and  a general  consensus  e.xisted  that  a taxonomy 
of  software  systems  was  of  considerable  practical  value.  From  other  considerations 
(see  K.  W.  Kolence's  paper  for  the  workshop),  the  session  chairman  believes  this  tax- 
onomy should  be  based  purely  on  structure,  not  function.  Further,  any  research  directed 
to  this  goal  should  be  fully  cognizant  of  the  many  uses  to  which  a successful  taxonomy 
would  be  put. 

.\lthough  the  topic  was  not  fully  discussed  within  the  group.  It  is  the  session  chairman’s 
belief  that  Dr.  Maurice  Halstead’s  "software  science"  work  and  the  work  by  Belady  and 
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Lehman  on  the  dynamics  of  evolution  of  software  are  sufficiently  supported  with  empiri- 
cal data  to  be  seriously  considered  in  any  research  on  the  general  topic  of  Rayleigh 
curves.  They  address  similar  questions,  and  appear  applicable  to  as  wide  a variety 
of  problem  areas.  If  all  these  approaches  continue  to  be  supported  by  empirical  data, 
then  a strong  presumption  e.xists  that  a synthesis  may  well  produce  impressive  and 
immediately  practical  results. 

A third  set  of  anomalies  and,  in  the  opinion  of  the  session  chairman,  weaknesses  exist 

in  the  material  presented  by  Putnam,  though  not  Norden.  In  a sense  they  appear  trivial, 

and  in  a practical  sense  they  truly  are  for  current  use  of  Putnam's  ideas.  In  a more 

abstract  sense,  however,  they  are  important  if  Putnam's  work  is  to  be  extended  and 

integrated  with  other  empirically  based  organized  knowledge  such  as  Kolence's  software 

physics  and  Halstead's  software  science.  The  problem  is  the  use  of  certain  terms  and, 

more  especially,  a consistent  use  of  appropriate  dimensional  units.  (This  criticism 

applies  equally,  in  the  session  chairman's  opinion  only,  to  Halstead's  work.)  Putnam 

2 

says  the  factor  K f t has  units  of  power  (work  per  unit  time)  and  K ^ t , units  of  force. 

d 

This  is  an  incompatible  statement.  Putnam  also  says  that  a system  has  an  "inherent 

2 

difficulty  factor"  represented  by  K f t^  and  at  the  same  time  says  that  this  appears  to 
change  somewhat  with  what  group  is  performing  the  work.  In  essence,  this  was  dis- 
cussed earlier  in  the  report,  but  the  point  being  made  is  that  the  statement  and  the  name 
"difficulty  factor"  are  incompatible.  -Again,  strictly  in  the  session  chairman's  opinion, 
a more  rigorous  approach  to  accuracy  in  statements  and  dimensions  is  needed  in  this 
work  now,  before  this  material  is  spread  too  widely  and  these  deficiencies  can  no  longer 
be  corrected. 

The  other  major  point  considered  by  the  group  was  the  question  of  dynamic  behavior 
analyses  of  the  Rayleigh  equation  in  its  present  form.  Before  turning  to  it,  however,  it 
should  be  emphatically  stated  that  the  criticisms  above  should  be  considered  small 
in  comparison  to  the  stature  of  the  actual  achievements  presented  by  Putnam  and  Norden, 
It  was  the  group  consensus  that  the  work  was  outstanding  and  of  Immediate  practical 
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value  in  current  efforts.  This  was  reflected  in  the  major  recommendations  of  the  full 
group  to  the  workshop. 

Spruce  Riordon's  paper  on  the  dynamic  behavior  of  a system  with  the  Rayleigh  equation 
characteristics  was  well- received  by  the  group.  Due  to  a shortage  of  time,  there  was 
little  in  the  way  of  a critical  discussion  on  the  subject  matter.  Within  these  limits,  how- 
ever, the  group  clearly  felt  that  the  dynamics  and/or  transient  behavior  of  Rayleigh  curve 
systems  should  be  investigated.  The  difficulty  of  course  would  be  how  to  gather  empiri- 
cal data  to  validate  any  mathematical  results  that  are  obtained  though  the  Belady-Lehman 
work  may  represent  a significant  step  in  this  direction.  The  group  consensus  was  that 
such  investigations  represent  a potentially  fruitful  research  area.  However,  the  personal 
opinion  of  the  session  chairman  is  that  this  will  only  be  true  if  careful  attention  is  paid 
to  the  problems  of  empirical  verification  from  the  very  outset  of  the  effort. 

Rav  leigh  curves  probably  do  represent  a ''natural"  behavior  of  a design  group  on  a design 
problem,  and  of  similar  groups  with  similar  problems.  That  is,  given  a large  number 
of  people  assignable  to  the  task,  the  amount  of  effort  actually  required  and  the  time  span 
of  such  effort  does  appear  to  reflect  some  natural  laws  of  design  and  the  design  process. 
•As  pointed  out  by  Elliott,  Miller,  and  Schwartz,  the  real  world  often,  if  not  normally, 
does  not  have  a large  number  of  assignable  people  for  a task.  As  a result,  manpower 
loadings  are  often  flat  or  trapezoidal,  and  may  or  may  not  be  sensitive  to  the  "natural"  - 
loading  requirements  of  the  effort. 

While  a fair  amount  of  discussion  was  held  on  this  subject  within  the  group,  little  in  the 
way  of  positive  suggestions  emerged.  In  the  session  chairman's  opinion,  this  may  have 
been  due  to  the  fact  that  we  failed  to  ask  the  right  questions;  e.  g. , in  a constrained  world, 
what  are  the  best  manpower  strategies  to  obtain  the  best  productivity  from  a limited 
number  of  people.  This  of  course  may  not  be  a "right  question"  either,  but  it  certainly 
would  have  led  to  a more  productive  discussion.  The  problem  should  be  faced  more 
directly  than  has  apparently  been  the  case  from  the  data  and  presentations  available, 
and  again  may  well  represent  a fruitful  research  direction. 
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The  recommendations  of  the  group  were  proposed  and  discussed  within  an  hour's  time, 
and  so  car..:ot  be  presented  as  carefully  weighed  and  precisely  stated.  Indeed,  much  of 
the  Session  Report  discussion  has  been  for  the  purpose  of  placing  these  recommendations 
in  a proper  ,'ontext.  I have  taken  the  liberty  of  injecting  some  personal  opinions  and  com- 
ments into  that  discussion,  but  only  when  these  points  had  not  been  brought  out  in  group 
discussion  due  to  the  limited  time  available  for  study  and  reflection. 

Even  with  a recognition  of  the  somewhat  hasty  nature  of  the  group  efforts,  it  should  be 
recognized  that  a remarkable  unanimity  of  opinion  existed  on  the  value  of  Putnam's  and 
Norden's  work,  and  on  the  need  for  efforts  such  as  proposed  by  Riordon.  As  a result, 
the  recommendations  given  below  should  be  considered  as  important  general  statements, 
arguable  perhaps  in  the  letter  but  not  in  the  spirit  of  what  they  say. 

The  general  recommendations  of  the  group  were  as  follows: 

1.  The  group  recommends  that  the  Army  Computer  Systems  Command  use  the  Rayleigh 
curve  techniques  as  developed  by  Putnam,  Norden,  and  others  as  a predictive  and 
control  tool  for  software  systems  development.  In  particular,  it  is  recommended 
that  a continuing  test  of  the  validity  of  these  techniques  in  CSC  projects  be  made 
over  as  wide  a range  of  project  size  and  content  as  possible.  .-Vt  the  same  time, 
other  useful  predictive  techniques  such  as  those  of  Belady  and  Lehman  should  be 
similarly  applied,  evaluated  and  developed. 

2.  The  group  recommends  research  into  the  underlying  ''phenomenology  or  "physics" 
of  the  processes  and  structures  which  give  rise  to  the  observed  behavior.  The  pur- 
pose of  this  research  is  to  understand  the  causes  and  possible  trade-offs  in  design 
complexity,  manpower,  and  time-to-complete  such  efforts  so  as  to  improve 

the  process.  Where  possible,  such  research  should  be  compatible  with  other 
empirically  based  "phenomenology."  Both  static  and  dynamic  system  behavior 
should  be  studied. 
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3.  The  group  recommends  that  the  Army  Computer  Systems  Command  begin  to  develop 
methods  and  procedures  to  implement  presently  available  life  cycle  curve  knowledge 
and  understanding  for  purposes  of  both  original  estimates  on  project  manpower  and 
completion  times  and  for  monitoring  actual  progress.  It  is  suggested  these  be 
tested  on  several  projects  for  accuracy  and  appropriateness,  simplified  where 
feasible,  and  integrated  into  the  appropriate  labor  and/or  cost  collection  systems 
for  routine  use  in  subsequent  projects. 

4.  The  group  believes  many  other  potential  and  practical  applications  of  these  con- 
cepts may  exist,  especially  when  combined  with  other  emergent  "phenomenologies. " 
As  such,  it  recommends  appropriate  research  and  consideration  of  these  concepts 
in  other  problem  areas  of  the  overall  life  cycle  management  process. 

Measurement  of  Quality  and  Reliability 
Chairman:  Robert  McHenry 

Robert  McHenry  presented  the  results  of  his  session  in  the  form  of  six  specific  recom- 
mendations. These  were  to  collect  data,  use  and  test  existing  models,  characterize 
the  problem  environment,  characterize  the  effects  of  change,  characterize  integration 
effects,  and  characterize  quality.  In  addition,  topics  were  noted  which  should 
receive  attention  at  subsequent  forums  addressing  SLCM  problems.  Details  of  these 
recommendations  follow. 

Recommendation  1:  Collect  Data 

Impose  mandatory  data  collection  on  Army  in-house  and  contracted  software  projects 
to  support  intra-project  reliability  analysis  and  reliability  research. 

As  a prerequisite  to  large-scale,  mandatory  data  collection,  clear  definitions  of  all 
data  items  must  be  developed.  First-cut  definitions  appear  in  recent  reports  prepared 
for  RADC.  The  participants  identified  the  following  data  requirements: 
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• ’’Formal"  reporting,  i.e. , failures  found  in  baselined  products  requiring  formal 
problem  reporting  applicable  to  operation  as  well  as  development  and  test 

calendar  time 
execution  interval 

process  phase  (requirements,  operation,  etc.) 

priority  or  severity 

qualitative  description  (taxonomy) 

• detection  (e.g. , review,  inspection,  execution,  etc.) 

• failure 

• code 

was  failure  induced  as  result  of  correction 
did  failure  uncover  similar  errors  (how  many) 
resource  consumption;  labor,  computer 

• detection,  e.  g.  test  team 

• diagnosis  (hardware  vs.  software  vs.  person) 

• correction 

identification  of  modules  fixed 

if  closed  — when  fixed  or  decision  not  to  fix 

Note:  open  failure  maintained  if  associated  errors  cannot  be  diagnosed 

• Equivalent  information  for  changes  to  baselined  products 

• "Informal"  reporting,  i.e.,  failures  found  in  unbaselined  products  (and  not 
subject  to  problem/change  control  in  CM/QA  terms), e.g. , library  updates  may 
estimate  failures. 

Recommendation  2:  Use  and  Test  Existing  Models 

There  already  exist  several  interesting  models  for  software  reliability  measurement. 
These  should  be  used  and  tested  against  historical  and  current  project  data.  Future 
work  should  concentrate  on  achieving  the  following  properties: 

• Simplicity 


• Informativeness 


• Appropriateness 

• Practicality 

• Wide  applicability.  j 

This  might  be  done  by  addressing  the  following  questions: 

• What  failure  models  are  "natural"  for  software? 

• How  can  costs"  and  "utilities"  be  incorporated  to  achieve  specified  reliability  ^ 

levels  ? ' 

1 

1 

• How  do  we  integrate  hardware  and  software  reliability  models  into  a model  for  I 

system  reliability? 

• Can  knowledge  of  program  structure  be  used  (e. g. , modules)?  ! 

i 

• What  are  the  common  properties  and  differences  of  existing  models  ? ^ 

The  state  of  the  art  is  such  that  no  models  can  be  definitely  chosen  in  preference  to  | 

others.  Parallel  and  competing  work  in  the  above  areas  is  desirable.  * 

I ' 

I Recommendation  3:  Characterize  Environment  } 

Initiate  research  studies  to  investigate  the  characterization  of  the  input  space  of  a pro- 
gram and  how  changes  in  the  input  space  can  affect  the  operational  reliability  of  the 
program. 

To  improve  reliability  modelling  and  reliability  prediction  by  characterizing  features  of 
' e.xecution  environments  which  Influence  the  observed  rate  of  failure. 

Develop  automatic  recording  to  be  built  into  future  software  so  that  the  type  of  usage 
causing  failure  can  be  identified. 

Identify  some  qualities  in  the  execution  environment  which  affect  the  observed  rate  of 
failure  of  a software  system.  It  should  be  used  as  inputs  for  a prediction  of  the  relative 
failure  rates  of  different  test/execution  regimes. 
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Current  reliability  models  assume  with  little  justification  that  the  reliability  information 
collected  before  a st'stem  goes  live  can  be  used  to  predict  subsequent  operational  per- 
formance. 

Develop  a characterization  of  the  components  of  stress  than  can  be  applied  in  testing 
software,  e.g. ; 

• Input  stress  - rate,  volume 

• Algorithm  stress  - near- singular  cases 

• Reduction  stress  - singular  cases,  input  errors. 

Develop  strategies  for  software  stress  testing. 

Recommendation  4:  Characterize  Change 

Determine  how  changes  are  reflected  in  the  reliability  models.  Identify  rate  of  change 
metrics  which  can  be  used  to  describe  the  nonsmoothing  aspects  of  reliability  change. 

Software  changes  reflect  changes  in  requirements/specifications.  These  changes  might 
be  treated  in  various  ways: 

• May  be  handled  quite  adequately  by  simply  continuing  to  use  existing  models 

• May  imply  a new  program  on  which  reliability  estimates  must  be  made 

• May  imply  restarting  the  testing  process  and  generating  reliability  data. 

The  questions  concerning  absorbing  new  data  by  already  existing  probability  distribu- 
tions are: 

• Are  there  specific  discontinuity  points  at  which  such  absorption  is  reasonable? 

• Can  we  treat  such  distributions  as  a collection  of  distributions? 

The  characterizing  of  change  is  important  to  change  introduction  strategies  to  assess 
the  extent  of  reasonable  change  and  to  determine  blocking  strategies  for  program 
releases. 

Recommendation  5:  Characterize  Integration 

Determine  the  effect  of  integrating  modules  on  existing  reliability  models  which  treat 
only  one  program  (module  or  system). 
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Collect  Hata  on  interface  errors  and  determine  whether  errors  are  intramodule  or  inter- 
face errors.  The  latter  may  include  all  combinations  of  software-hardware-human  inter- 
faces. 

Developed  models  which  relate  software  error  incidence  to  cost  of  integrating 
previously- tested  and  new  software. 

Recommendation  6:  Characterize  Quality 

Determine  the  measurable  attributes  of  software  quality. 

Traditional  quality  assurance  may  be  paraphrased  as  conformity  assurance.  Specifica- 
tion conformity  is  not  adequate  to  assess  several  software  characteristics  associated 
with  quality  such  as  maintainability  and  portability.  The  quality  characterization 
research  includes  specification  and  realization  techniques  in  addition  to  attribute  defini- 
tion, Some  first-cut  definitions  and  approaches  have  been  prepared  under  government 
contracts. 

Additional  Topics 

Several  topics  of  importance  to  this  group  were  not  discussed  because  of  lack  of  time. 
Foremost  among  these  topics  were: 

• Maintainability  - techniques  and  practices 

• Maintainability  - definitions 

• Maintainability  - mathematical  models 

• .Availability  - definitions 

• Availability  - mathematical  models 

• System  reliability  - including  hardware,  software  and  human  operator 

• Acceptance  tests  - as  stand-alone  procedures,  and  as  they  pertain  to  reliability 
models. 

The  reader  should  not  infer  that  there  is  a lack  of  preliminary  work  or  increase  of 
Importance  in  these  areas.  In  fact,  should  this  discussion  resume  in  the  future,  such 
topics  would  be  first  on  the  agenda.  Continuing  work  in  these  areas  is  desirable. 
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Human  Factors,  Individual  and  Group  Productivity  Measures 
Chairman:  Professor  Maurice  Halstead 

Session  Overview 

In  an  industry  as  labor-intensive  as  our  own,  the  measurement  of  both  individual  and 
group  productivity  is  a vital  concern.  The  application  of  knowledge  gained  from  these 
measures  to  real-world  problems  is  certainly  of  equal  concern.  The  participants 
in  the  "Human  Factors,  Individual  and  Group  Productivity  Measures"  sessions  dealt 
with  the  following  general  areas  of  interest: 

• The  basis  for  Software  Engineering 

• Measures  of  productivity-metrics 

• Measurement  techniques 

• Application  of  measurement  results. 

One  of  the  major  outcomes  of  this  session  was  a realization  of  the  synergistic  relation- 
ship of  each  participant's  area  of  endeavor  to  those  of  the  other  participants.  For 
example,  experimental  data  collected  by  Dr.  Love  and  actual  project  data  collected  by 
Dr.  Walston  were  further  supporting  evidence  for  Dr.  Halstead's  Software  Science. 

} 

The  sessions  were  quite  orderly  and,  for  the  most  part,  consisted  of  the  presentation  I 

of  the  SLCM  papers  with  interaction  among  the  participants  from  time  to  time  on  speci- 
fic issues  raised  by  the  presentation.  The  meat  of  the  presentations  can  be  found  in  the 
papers  themselves.  What  follows  is  a very  brief  summary  of  each  presentation  and  a 
final  section  listing  the  problem  areas,  promising  research  areas,  and  research  objec- 
tives identified,  or  more  appropriately  distilled,  by  the  participants  durit^  the  final 
session. 

Presentation  Summaries 

I 

Dr.  Halstead 

I 

Dr.  Halstead's  presentation,  "Potential  Impacts  of  Software  Science  on  Software  Life 
Cycle  Management,"  pointed  out  that  Software  Engineering  must  have  a natural  science 
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base  to  be  a "quasi-complete”  engineering  discipline.  Halstead's  "Software  Science" 
might  be  considered  as  one  candidate.  "Software  Science"  is  based  on  a few  language 
independent  parameters  which  can  be  easily  (even  automatically)  measured.  The 
derivation  of  Halstead's  E was  shown  along  with  supporting  evidence  of  its  accuracy. 

A number  of  applications  concerning  software  life  cycle  management  were  mentioned 
and  specific  areas  for  study  indicated. 

Dr.  Love 

Dr.  Love's  presentation,  "Software  Psychology:  Shrinking  Life  Cycle  Costs,"  also 
states  that  Software  Engineering  must  have  an  underlying  body  of  scientific  knowledge 
and  that  the  disciplines  of  psychology  and  statistics  may  help  add  to  that  knowledge.  To 
understand  and  successfully  apply  any  new  software  engineering  technique,  we  must 
know\^  it  is  better  than  another.  Software  Psychology  is  the  "application  of  the  knowl- 
edge and  technology  of  psychology  to  improve  the  productivity  of  programmers." 

The  results  of  three  completed  studies  were  presented.  The  first  study  represents  the 
first  time  Halstead's  theory  of  Software  Science  was  tested  with  appropriate  experimental 
controls.  A high  correlation  was  found  between  performance  in  the  experiment  and 
Halstead's  E.  Other  studies  "Individual  Differences  in  Programmer  Productivity"  and 
"Critical  Factors  in  Software  Development"  provided  some  interesting  and  some  counter- 
intuitive results. 

Some  e.xpected  results  of  this  work; 

• A set  of  equations  to  help  evaluate  actual  techniques  relative  to  schedule,  cost 
and  productivity 

• The  relationships  among  techniques 

• The  identification  of  new  techniques  which  may  be  of  use. 

Dr.  Manley 

Dr.  Manley's  presentation,  "Computer  Systems  Engineering  Management  Education," 
concerned  the  education  of  practitioners  of  software  engineering.  The  premise  that 
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industr}'  roles  are  changing  and  traditional  educational  systems  no  longer  meet  their 
objectives  indicates  a new  approach  is  required.  Johns  Hopkins'  approach  involves  the 
blending  of  science  and  management  to  provide  a realistic  methodology  for  not  only  deal- 
ing with  but  improving  the  management  world. 

Dr.  Manley's  paper  gives  a complete  overview  of  his  approach  and  complete  descriptions 
of  the  Computer  Systems  Engineering  Management  courses  offered  at  Johns  Hopkins/APL. 

Dr.  Walston 

Dr.  Walston's  presentation  was  based  on  "A  Method  of  Programming  and  Estimation" 
by  E.  E.  Walston  and  C.  P.  Felix  published  in  IBM  Systems  Journal,  Vol  16,  No.  1. 

The  motivation  behind  the  establishment  of  the  programming  measurement  data  base  at 
IBM's  Federal  System  Division,  the  vehicles  used  to  take  measures,  and  current  and 
ongoing  work  were  presented. 

The  data  base  is  used  to  evaluate  new  hypotheses  and  improved  programming  techniques 
and  to  provide  Indirect  feedback  to  the  data  supplier  as  well  as  to  provide  a historical 
record  of  software  development  performed. 

The  actual  measurement  program  generally  starts  at  contract  award  and  concludes  at 
acceptance  or  cutover,  i.  e. , the  development  phase  only.  The  method  of  collecting 
data  was  discussed  and  some  preliminary  results  of  measurement  techniques  were  pre- 
sented. 

Dr,  Walston  showed  some  recent  analytical  results  with  the  data  in  his  article  support- 
ing Dr.  Halstead's  E Hypothesis. 

Raymond  Wolverton 

Mr.  Wolverton's  presentation,  '.'The  Cost  of  Developing  Large-Scale  Software"  described 
a "state-of-the-art,  " accurate  (n  10  percent  accuracy),  working  cost  estimation  algorithm 
usable  by  end  management.  The  methodology  based  on  a 25x7  structural  forecast  matrix 
(25  activities  x 7 phases)  was  presented. 


Actual  project  experience  was  also  discussed  and  a number  of  interesting  units  of  meas- 
ure were  discussed: 

• The  number  of  cards  a programmer  can  maintain  is  a more  meaningful  measure 
than  the  cost  to  change  a line  of  code  during  the  maintenance  phase 

• Basic  unit  of  measure  is  1000  machine  language  instructions 

• Errors  counted  per  month  per  1000  instructions. 

Dr.  Zelkowitz 

Dr.  Zelkowitz  presented  a short  overview  of  the  University  of  Maryland/NASA  Goddard 
Space  Flight  Center  Software  Engineering  Laboratory.  The  general  goals  of  the  facility 
are  to  monitor  and  analyze  the  production  of  software  at  Goddard.  Specific  goals  are  to 
organize  a data  base,  isolate  parameters  and  find  out  what  is  happening  as  opposed  to 
what  should  be  happening  during  software  production. 

Three  types  of  e.xperiments  are  being  conducted:  screening  - to  discover  the  range  of 
variables  (to  debug  methodology);  semicontrolled  - e.xtension  of  screening  methodology 
with  more  control;  and  controlled  - a further  extension  of  service-controlled  (use  of 
duplicate  projects). 

The  problem  of  consistency  across  projects  (transferable  data)  was  discussed.  This  is 
a general  problem,  .\mong  the  session  participants  one  counts  lines  of  code  (JCLand 
comments  included),  another  counts  machine  language  instructions,  and  a third  counts 
executable  FORTRAN  statements. 

Key  Items 
Problem  Areas 

• A standard  nomenclature  is  needed  — both  a basic  vocabulary  and  a set  of  com- 
mon definitions  is  a must 

• A general  need  for  more  data  — in  particular  data  that  is  transportable. 
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Promising  Experiemental  and  Theoretical  Research  Areas 
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• The  implementation  of  change 

• Computer  Systems  Management  Engineering 

r - Management  Science 

- Behavorial  Science 

- Team  Organization 

- Practical  Applications 

• Software  science. 

[ 

Research  Objectives 

• Establishment  of  a "quasi-complete"  engineering  science  base 

• Design  of  experiments 

i • Identification  of  erroneous  conclusions  due  to  incorrect  parameters. 
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On  Software  Engineering 

John  Staudhammer'* 

US  Army  Computer  Systems  Command 
Fort  Belvoir,  Va. 

Over  the  last  fourteen  months  I served  as  Technical  Advisor  to 
the  Army  Computer  Systems  Command.  During  this  time  1 had  occasion 
to  examine  a great  many  problems  in  the  production  of  responsive 
software  and  to  discuss  with  the  Commander,  Major  General  Jack  L. 

Hancock, his  views  on  software  engineering.  This  presentation  borrows 
heavily  from  his  ideas  and  his  published  speeches  (l). 

One  of  the  fxuictions  of  the  Technical  Advisor  was  to  chair  the 
Army's  Integrated  Software  Research  and  Development  (iSRAD)  Program 
council.  During  this  time  ISRAD  was  supported  by  a contract,  competi- 
tively won  by  Computer  Sciences  Corporation.  Two  major  significant 
actions  were  completed  with  this  support;  the  hosting  of  the  Army  Soft- 
ware symposium  of  May  1977  (2),  and  the  convening  of  this  workshop. 

ISRAD  has  also  become  a forum  for  ideas  in  software  engineering 

1 

problems,  the  most  pressing  of  which  is  the  adequate  prediction  and  ! 

management  of  costs.  With  Colonel  Putnam's  work  on  the  dynamics  of 
software  life  cycle  costs  (3i^)  feel  that  we  have  an  understanding 

} ■ 

of  the  enormity  of  this  problem  and  we  are  conviced  of  the  need  to  ‘ 

I 

further  explore  and  find  the  major  factors  in  this  matter.  We  are 

confident  that  better  methods  will  emerge  in  the  near  future.  ^ 

I 

i 

The  Army  Computer  Systems  Command  comprises  about  1300  people  | 

and  all  aspects  of  software  engineering  appear  in  the  command's  daily  ) 

i 

* On  leave  from  North  Carolina  State  University,  ] 


operations.  The  command  is  the  developer,  maintainer  and  disseminator 
of  all  financial,  personnel  and  logistics  programs  in  use  at  all  Army 
installations.  We  do  not  operate  the  software,  but  are  responsible  for 
delivering  operable  systems  and  must  respond  to  emergencies  that  may  arise 
in  executing  our  codes.  To  insure  uniformity  all  of  the  systems  software 
is  distributed  in  object  code  form  to  about  100  installations. 

The  work  of  the  command  is  in  creating  large  business  codes.  The 
average  program  size  is  300,000  lines  of  COBOL  code,  has  a life  of  some 
seven  years  and  costs  about  350  man-years.  The  design-time,  the  time  for 
first  release  after  official  funding,  is  2.7  years.  Our  average  coding 
rate  is  2000  statements  per  year,  giving  a gross  figure  of  about  $20  per 
statement,  allowing  a 100^  overhead.  Clearly  this  is  a large  operation 
and  a systematic  design  procedure  is  mandatory. 

One  of  the  first  actions  I was  involved  in  last  summer  was  the 
selection  of  a command-standard  procedure  for  program  design.  After 
a study  of  existing  methods  we  chose  the  Jourdzua-Constemtine  structured 
design  methodology.  Since  not  all  pOO  programers  could  switch  to  this 
method  overnight,  a phased  implementation  plan  was  agreed  upon;  also,  we 
decided  to  have  two  pilot  projects  to  demonstrate  the  efficacy  of  the 
method. 

One,  Project  VIC—  Visibility  of  In-transit  Cargo  (5)  — was  done  in 
relative  isolation  in  Europe.  This  system  allows  the  tracing,  routing 
and  re-routing  of  various  cargo  units  throughout  a theater  of  operations. 
The  original  user-supplied  requirements  were  carefully  reviewed  prior  to 
code  design.  At  that  time  the  required  effort  to  first  fielding  the  code 
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was  estimated  at  5000  man-hours.  Since  the  effort  was  a relatively 
modest  one  it  was  felt  that  little  risk  was  involved  in  trying  the 
newly  decided-upon  Structured  Design  Teclmology.  Tliroughout  the  design 
little  direct  effort  was  made  to  gather  special  data  on  the  performance 
of  the  design  team.  With  the  user  working  in  daily  contact  with  the 
program  designers  the  Detailed  Functional  Requirements  were  analyzed  and 
the  top-down  design  was  started.  Quickly  it  was  apparent  to  the  user- 
team  as  well  as  the  code-developer  team  that  the  stated  requirements 
were  grossly  inadequate.  Of  course,  both  parts  of  the  team  had  to  have 
the  same  background  training  and  had  to  use  the  same  tools. 

* 

With  the  inadequacy  of  the  requirements  accepted  by  the  user,  the 
effort  required  was  re-estimated.  It  is  now  at  about  15  man-years,  some 
six  times  the  original  guess.  The  tools  used  relied  on  a more  Imowledge— 
able  user  and  requirements  more  carefully  analyzed.  It  is  also  worth 
noting  that  the  user  continues  to  have  a requirements  analysis  group  about 
as  large  as  the  design  team.  The  net  result  so  far  has  been  a more  care- 
fully thought-through  design,  a user  who  now  has  far  more  detailed 
insight  into  his  own  business  and  a code  design  understandable  by  the 
eventual  user.  Aspects  of  the  system  have  been  uncovered  that  were 
unsuspected  earlier  in  this  work.  For  example,  security  aspects  were 
discovered  early  in  the  design  and  steps  for  adequate  safeguarding  could 
be  initiated  early;  without  the  systematic  design  process,  such  surprises 
often  necessitate  massive  code  changes  costing  sometimes  as  much  as  the 
original  "complete"  code. 

The  other  project,  the  Direct  Support  Standard  Supply  System  (DS-4) 
(6,7)  was  done  with  much  forethought  to  gathering  statistics  on  the 
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progress  of  the  design  team  so  that  the  efficiency  of  using  the 
structured  design  process  could  be  evaluated.  Compared  with  the  VIC 
project  the  design  was  conducted  in  the  limelight.  This  project  was 


a re— direction  of  an  earlier  design  effort  which  was  given  added 
and 

requirements  started  anew  in  June  1976.  The  target  date  for  proto- 
type code  was  Summer  1977.  Using  historical  data  on  similar  projects 
not  done  in  a structured  way  Colonel  Putnam  estimated  the  coding  pro- 
ject to  yield  30,000  lines  of  COBOL  code,  and  using  his  risk  analysis 
procedures  he  estimated  the  chances  of  successful  completion  at  about 
one  percent.  He  also  gave  50-50  odds  for  completion  in  18  months. 

The  latest  data  indicate  that  the  program  now  has  about  3000  lines  of 
Met»-C0B0L,  completed  about  1 June  1977,  with  "most"  of  the  system 
executed  to  completion  at  about  10  August  and  that  the  overall  program 
will  contain  about  30,000  lines  of  Meta-COBOL.  The  design  effort  has 
been  about  15  man-years  so  far  and  may  go  to  20  man-years. 

Thete  was  an  additional  requirement  placed  on  this  system:  although 
developed  on  an  IBM  370  system,  targeted  to  rtm  on  an  IBM  360/5O  machine, 
there  was  a requirement  to  demonstrate  it  on  a minicomputer.  Several 
approaches  to  this  conversion  were  taken;  one  of  these  converted  about 
10,000  lines  of  COBOL  (obtained  by  expanding  the  Meta-COBOL  source)  in 
about  600  man-hours  on  to  a PDP-II/70  system.  ViThile  a few  compromises 
had  to  be  made,  basically  programs  and  files  could  be  converted  with 
relative  ease  once  the  functions  of  the  code  were  well  understood  (S). 

The  presence  of  the  lead  programer  of  the  original  code  design  team  on 
the  conversion  team  was  of  course  a great  help,  but  probably  more 
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important  was  tUe  enforced  discipline  of  structured  design  concepts, 
of  meaningful  mnemonics,  adherence  to  ANSI  standard  COBOL  in  the  original 
source  and  readable  code.  However,  the  single  most  important  factor 
in  this  conversion  was  the  high  motivation  of  the  team  members. 

I- 

The  basic  conclusions  on  the  DS-4  exercise  so  far  are  that  code 
conversion  productivity  is  highly  dependent  upon  motivation;  that  there 
is  a great  need  for  functional  expertise  on  the  conversion  team,  and 
that  code  production  rates  are  about  the  same  in  structured  design 
as  in  more  conventional  methods.  There  is,  however,  ^ marked  difference 
in  the  ease  with  which  structiired  code  can  be  read;  this  indicates  a 
great  hope  for  reduced  maintenance  costs. 

Although  data  is  scanty,  the  code  production  rates  for  COBOL  and 
Meta— COBOL  seem  to  be  about  the  same.  This  in  turn  would  indicate 
that  efficiencies  in  creating  a ruiming  code  are  most  likely  to  come 
from  changes  in  source  language  rather  thzui  enforcing  structured  con- 
structs using  a given  language. 

Part  of  the  problem  in  comparing  two  higher-order  languages  is  the 
very  word  "productivity".  Basically  what  is  being  measured  is  not  clear  - 
for  example,  code  produced  with  many  inefficient  statements  could  well 
result  in  more  lines  of  code,  but  may  require  less  effort,  less  design, 
for  its  creation.  By  the  conventional  measures  of  "code  productivity" 
the  inefficient  program  would  be  judged  a better  product.  Probably  the 
only  true  measure  of  a programer's  affectiveneas  is  the  rate  at  which 
he  reduces  system  requirements  to  checked-out  code.  But  since  these 
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requirements  are  all  quite  different,  no  jjood  objective  measures  exist 
for  measuring  programer  effectiveness.  This  of  course  should  not  be  a 
surprise,  for  there  are  no  objective  measures  for  creative  activities  in 
any  field.  Recognizing  this  fact  we  should  not  delude  ourselves  by  citing 
code  production  numbers  a measiures  of  effectiveness.  Possibly  software 
quality  measures  eventually  may  shed  some  light  on  this  subject. 
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The  problem  of  code  portability  is  not  unique  to  the  Computer 
Systems  Command.  We  do  have,  however,  a peculiar  environment  where 
we  are  forced  to  produce  and  maintain  software  systems  which  will  con- 
currently be  executed  on  various  alien  mainframes.  Part  of  the  problem 
is  to  deliver  checked-out  codes  to  dissimilcir  equipments.  One  project 
developed  a single  source  code,  maintains  that  one  source  and  produces 
from  it  different  target  object  codes.  In  this  case  IBM  370  code  was 
in  existence  and  undergoing  prototype  evaluation  when  an  additional 
requirement  arose  to  transport  the  software  to  selected  other  large 
machines.  Because  the  system  was  still  in  development,  it  was  subject  to 
substantial  changes  to  satisfy  new  or  revised  functional  requirements. 
Demands  for  final  systems  availability  of  all  versions  required  con- 
current parallel  efforts  - that  of  continuing  the  basic  system  develop- 
ment and  modification  - and  that  of  developing  two  additional  versions 
for  CDC  6300/6600  and  Univac  IIO6/IIO8.  A team  developed  the  new  versions 
while  the  original  Application  System  Developer  (ASD)  continued  his  work 
on  the  prototype.  A copy  of  the  current  IBM  baseline  was  transferred 
to  the  team  for  its  use.  Any  subsequent  changes  made  to  the  baseline 
by  the  developers  were  uniquely  identified  in  the  source  data  sets. 


Following  the  initial  multi-version  development  and  validation  of  all 
three  hardveire  versions  against  the  initial  baseline,  the  multi-version 
source  programs  were  updated  with  the  system  changes  which  had  been 
implemented  by  the  developer  and  again  validated.  The  multi-version  system 
software  package  was  then  placed  into  continuing  mainteneance  (9,10). 


Both  the  Structured  Design  and  the  single-source,  multi-teurget  code 
are  efforts  to  anticipate  changes  in  the  way  we  will  have  to  do  business 
in  the  future:  we  foresee  a multiplicity  of  hardware  which  must  be 
accomodated  in  the  various  Army  installations. 


One  of  the  research  areas  of  deep  concern  is  the  assessment  of 
factors  in  portability  of  software,  particularly  large  COBOL  source 
code.  It  is  well  understood  that  language  standards,  implementation 
differences  and  degree  of  adherence  to  structured  design  constructs 
all  have  a great  impact  on  the  ease  with  which  programs  can  be  transferred. 
A recent  study  focused  on  these  and  related  questions  in  the  framework 
of  current  practices  (ll).  While  questions  of  data  portability  have 
not  been  resolved,  the  recommendations  include  using  only  a subset  of 
ANSI  COBOL  consistent  with  the  implementations  on  all  target  machines, 
the  use  of  a translator  to  replace  vendor-dependent  syntax,  specialized 
Meta-COBOL  macros  which  will  expand  to  specific  vendors'  language  con- 
structs and  the  avoidance  of  specialized  executive  software. 


This  effort  was  a recognition  on  this  agency's  part  that  systematic 
research  and  development  work  had  to  be  undertaken  in  all  aspects  of 
software  engineering  to  aid  in  developing  methods  for  more  responsive 
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software.  A comprehensive  five-year  reseeirch  and  development  plan  was 
formulated  and  published  (12).  It  is  intended  that  this  plan  be  updated 
and  modified  as  necessary  and  thus  to  be  made  into  a "living  document". 
Several  hundred  copies  have  been  distributed  to  other  Department  of 
Defense  agencies,  industry  and  universities;  it  is  hoped  that  this  plan 
will  form  the  basis  of  research  proposals  as  well  as  proposals  for 
changes  of  the  plan  itself. 

While  we  are  waiting  for  all  the  steps  in  the  establishment  of 
far  more  effective  software  engineering  methods  to  be  researched, 
evaluated  and  implemented,  we  may  outline  some  of  the  more  likely 
future  solutions.  One  of  our  most  serious  shortcomings  is  the  in- 
ability to  forecast  development  efforts  and  another  is  the  failure  to 
produce  satisfactory  software.  Generally  more  and  more  system  failures 
are  being  attributed  to  the  fact  that  the  potential  system  user  does 
not  state-  his  requirements  clearly,  adequately  , in  sufficient  detail, 
in  cui  understandable  language,  so  that  the  system  may  be  properly 
designed.  The  literature  has  pondered  this  problem  and  the  general 
thrxist  has  been  twofold.  The  first  thrust  is  for  tools  that  designers 
cem  use  to  compensate  for  poor  user  requirements,  and  the  second  is 
aimed  at  how  to  get  the  user  to  specify  his  needs  in  a "clear,  consistent 
and  unambiguous  statement  which  can  serve  as  a basis  for  the  follow-on 
design  specification." 

However,  tools  to  compensate  for  poor  user  requirements  do  not 
substitute  for  having  the  user  have  a clear  understanding  of  his  own 
needs.  For  twenty-five  years  we  have  been  after  the  user  to  specify 


his  requirements  in  a complete,  consistent  and  unambiguous  manner  with- 
out success.  We  must  therefore  conclude  that  in  many  cases  it  is  im- 
possible for  him  to  do  so.  Providing  only  us,  the  program  developers, 
with  better  requirements  analysis  tools  is  only  a minor  help.  In  the 
design  of  management  information  systems  we  must  be  responsive  to  the 
manager's  needs.  However,  management  decision  making  is  not  a science, 
hut  an  art.  In  fact,  it  is  , as  one  author  stated,  not  one  art,  but  two: 
the  art  of  making  decisions  based  on  inadequate  information,  and  the  art 
of  living  with  the  results  of  those  decisions. 

Management  is  managing  change.  The  abnormal  situation  is  the  status 
quo  which  the  manager  has  to  overcome  in  order  to  get  back  to  the  normal 
task  of  causing  and  manging  chjuige.  If  one  can  identify  the  decisions 
that  a manager  has  to  make,  identify  the  criteria  by  which  he  makes  those 
decisions,  put  a value  on  the  resulting  courses  of  action,  and  hold  this 
value  system  constant,  there  would  be  little  need  for  highly  trained  and 
qualified  managers.  In  fact,  we  may  require  fewer  computers  as  well. 

But  mangers  can  not  identify  the  decisions  which  they  will  have  to  make, 

can  not  identify  the  criteria  they  use,  can  not  quantify  the  criteria, 

and  can  not  put  a value  on  the  resultant  courses  of  action;  and  they 

can  certainly  not  express  them  in  "a  complete,  consistent  and  unambiguous 

requirements  statement  which  can  serve  as  the  basis  for  design  specifications' 

And  above  all,  there  is  no  way  that  mangers  can  decree  holding  requirements 

constant. 

In  asking  a manger  to  define  his  requirements,  we  must  recognize  that 
he  can  respond  only  with  experiences  and  knowledge  which  he  already  has. 
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Since  the  system  he  is  asking  us  to  develop  is  to  operate  in  the  future 
we  must  introduce  him  to  data  processing  ideas,  solutions  and  techniques 
which  are  likely  to  be  used  in  the  future,  and  must  introduce  these 
ideas  to  him  during  the  process  of  requirements  definition  so  that  he 
may  make  reasonable  choices  about  his  evolving  system.  We  are  some- 
what in  the  role  of  a restaurant  specializing  in  Antarctic  food:  although 
we  may  have  a menu,  its  entries  are  likely  to  be  meaningless  to  our 
potential  customers,  they  are  unlikely  to  know  the  taste  characteristics 
and  we  will  have  to  be  ready  to  describe,  explain  and  demonstrate  on 
occasions.  The  same  ought  to  be  true  for  the  process  of  devising  new 
computer-based  systems.  But  the  almost  universal  mode  of  systems  design 
is  to  demand  of  the  user  his  requirements  in  a format  which  is  complete, 
not  over-constraining  on  we,  the  developers,  but  which  is  clear,  con- 
cise and  unambiguous.  And  we  want  those  requirements  on  time.  Also 
under  no  circumstances  must  the  user  tell  us  how  to  do  the  Job. 

Straight Jacketing  him  to  such  an  extent  is  problem  number  one. 

All  of  us  are  aware  of  the  statistics  on  the  at  least  a hundred- 
fold effort  required  to  make  changes  to  a program  after  the  system  has 
become  operational  compared  to  making  changes  during  development  (13) • 
Also  we  are  aware  of  the  ratio  of  systems  development  effort  to  system 
maintenance  effort.  At  the  Computer  Systems  Command  the  maintenance 
effort  is  between  73  and  80  percent  of  the  total  life  cycle  cost. 

Of  course,  maintenance  does  not  mean  simply  to  correct  errors.  The 
vast  majority  of  the  work  is  concerned  with  putting  the  system  in  a 
condition  that  the  user  "really  wanted"  in  the  first  place. 


Here  we  might  note  the  lack  of  basic  knowledge  in  our  teclmology. 
While  we  eire  all  horrified  by  the  high  cost  of  patching  an  error  in 
delivered  code,  let  me  point  out  that  one  way  to  lower  the  per-error 
cost  is  to  deliver  even  faultier  code  in  the  first  place.  Then  the 
fixing  would  be  averaged  over  a larger  number  of  errors  thus  possibly 
improving  the  cosmetics  of  the  statistic.  Conversely  consider  a utopiem 
code  performing  as  required  after  a single  error  was  corrected  halfway 
through  the  life  cycle.  Obviously  the  software  had  to  be  maintained 
with  some  resources  devoted  to  studying,  reading,  testing  the  code 
used  by  the  "customers".  The  entire  cost  of  this  activity  would  have 
to  be  charged  against  a single  error. 

The  entire  field  of  software  engineering  suffers  from  such  lack  of 
basic  data,  of  basic  insight.  We  may  have  many  measures,  but  do  not 
know  the  invariants  of  the  field:  we  are  still  waiting  for  the  first 
orderly  analysis  which  will  define  the  analogs  of  the  Mach,  Froude  and 
Weber  numbers  which  moved  fluid  dynamics  from  an  art  to  a science. 

Returning  to  the  discussion  of  user  requirements  and  our  need 
to  aid  users  in  formulating  their  needs,  it  seems  obvious  that  we 
could  profitably  concentrate  our  attention  on  developing  concepts, 
tools  and  techniques  which  enhance  the  user's  ability  to  specify  his 
needs  early,  but  in  such  a way  that  recognizes  his  lack  of  full  know- 
ledge and  assists  him  in  redefining  and  refining  those  requirements 
which  he  stated  initially.  This  redefinition  would  be  based  on  an 
early  assessment  of  the  likely  effects  of  meeting  the  requirements  as 
originally  specified.  Such  an  approach  would  greatly  reduce  costly 
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systems  maintenance  and  reduce  the  time  required  to  have  operationally 
effective  systems. 
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Robert  A.  Frosch,  past  Assitant  Secretary  of  the  Navy  for  Research 
and  Development  pointed  out  that  "anyone  who  has  ever  done  a develop- 
ment or  a design  as  opposed  to  setting  up  a management  system  for  doing 
so,  is  veil  aware  of  the  fact  that  the  real  world  proceeds  by  a kind 
of  a feedback  process  which  looks  more  like  a helix  than  a line" . Hence 
we  can  state  Problem  Number  Two  as  the  lack  of  a meaningful  feedback  to 
the  user  in  a timely  enough  fashion  to  allow  him  to  monitor  our  under- 
standing of  his  problem. 

Thus  we  need  to  re-orient  our  thinking  in  two  ways; 

1.  to  recognize  that  the  initial  user  requirements  will 
typically  not  be  true  reflections  of  what  the  user  actually 
needs  and  that  his  final  requirements  will  come  only  after 
many  changes  will  be  made.  Just  the  recognition  of  this 
fact  would  be  a major  improvement  in  our  business. 

2.  To  build  a means  by  which  we  speed  up  the  process  for 
inexpensively  "programing"  the  initial  user  requirements 
and  furnishing  the  results  in  such  a way  as  to  facilitate 
making  further  refiiements.  This  process  would  be  repeated 
many  times. 

It  sounds  like  heresy,  but  it  may  be  more  effective  for  the  user  to  take 
far  less  time  to  define  his  "requirements"  initially  than  he  does  now. 

These  "half-specified"  requirements  would  then  be  given  to  the  design 

team  early  to  be  used  in  a first  cut  at  a product.  The  output  from  the  | 

first  cut  would  be  a familiarization  with  the  requirements.  Gaps  in  the 
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requirements  would  be  filled  in  by  tlie  user  on  a demand  basis. 

In  support  of  this  way  of  system  design,  we  need  a conceptual  change 
in  the  way  we  do  business  and  much  additional  research  and  development 
leading  to  two  classes  of  tools.  The  change  in  concept,  which  must  be 
accepted  by  both  the  designer  and  user  communities,  would  be  explicit 
recognition  of  the  essentiallity  of  the  designer  becoming  more  intimately 
familiar  with  the  functional  areas  in  which  he  is  working  and  of  a change 
in  the  way  reqirements  are  developed.  The  familiarization  would  have 

to  come  from  a direct  interchange  between  the  designer  and  the  user  as 

* 

well  as  through  formal  training.  The  change  in  preparation  of  the  i 

requirements  would  involve  the  user  preparing  his  initial  requirements 
in  a gross  form,  furnishing  these  to  the  system  designers,  who  in  turn 
would  work  directly  with  the  user  in  putting  these  in  a form  from 
which  the  designer  could  prepare  a simiilation.  Thus  the  first  class 
of  tools  which  must  be  developed  are  simulators.  Simulations  were  used 
in  the  past  for  many  purposes,  but  for  the  most  part  they  were  used  for 
providing  information  on  probable  system  runtimes,  queue  times  and  . 
lengths,  peak  loading,  and  the  like.  We  have  not  exploited  the  capa- 
bilities of  simulators  to  produce  answers  to  many  of  the  basic  questions 
that  system  users  ought  to  be  asking. 

For  example  in  an  Inventory  control  system  obviously  we  need 
certain  basic  inputs:  requests  for  supplies,  receipt  information, 
inventory  data,  and  data  on  cancellations.  The  user,  typically  operating 
in  an  area  of  less  than  optimum  information,  will  probably  specify  many 
other  inputs  which  the  system  generates  as  an  output  at  some  earlier 
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time.  For  example  there  mij;ht  be  speuilications  on  back-ordering; 
information,  substitutions,  and  the  like.  When  the  user  specifies 
such  data  in  the  requirements  statement  he  has  no  way  of  knowing  the 
volumes  of  output  and  interactions  of  these  transactions  with  one  another. 
The  results  today  are  often  the  functional  systems  which  generate  paper 
so  profusely  and  choke  themselves  on  their  own  files  and  reports. 

If  the  simulation  tools  are  well  developed  and  understood,  the 
production  of  the  simulations  would  not  have  to  be  terribly  expensive. 
Based  on  the  results  of  the  early  simulations  the  user  would  be  in  a 
position  to  make  impact  emalyses  in  terms  of  cost,  time,  work-load  and 
assess  the  impact  on  all  the  system's  users.  In  the  truest  sense  users 
include  everyone  who  comes  in  contact  with  the  system  either  through 
inputting  to  it,  receiving  outputs  from  it,  or  interacting  through 
intermediate  transactions.  The  simulation  and  modeling  envisioned  here 
would  permit  the  assessment  of  the  system  on  all  its  users. 

After  the  simulations  have  been  evaluated  by  the  user  and  the 
requirements  have  been  re-defined  and  purified  through  a process  of 
iteration,  the  next  step  would  be  the  production  of  a "breadboard"  of 
the  key  modules  of  the  system.  The  term  "software  breadboard"  comes 
from  the  concept  of  hardware  breadboards.  One  of  the  objections  to  the 
analo(pr  between  these  two  breadboards  has  been  the  tacit  understanding 
that  once  a software  breadboard  is  produced  it  is  the  final  deliverable 
product.  Unlike  the  equipment  breadboards,  which  are  used  to  make 
decisions  regarding  design  choices  and  future  production,  we  are 
accustomed  to  thinking  of  software  as  being  fully  usable  once  coded. 
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la  the  context  used  here  the  software  breadboard  is  a product 
developed  with  little  or  no  thought  to  programming  efficiency,  to 
computer  run-time,  or  cost  effectiveness  of  the  breadboard  itself. 
Furthermore,  software  bread-boards  would  be  produced  for  only  key 
modules  of  a system.  For  example,  in  an  inventory  control  system, 
bread-boards  might  be  produced  only  for  the  edit  modules,  the  daily 
cycle,  and  report  output  modules  — not  for  the  n^iad  of  other 
modules  involved.  The  bread-board  would,  not  be  a final  specification, 
but  would  be  used  by  the  designer  and  the  user  to  certify  to  the 
validity  of  a processing  concept  and  as  the  basis  for  producing  the 
final  design  specification.  The  bread-board  would  be  "throw  away" 
coding  in  all  respects.  This  code  is  similar  to  an  engineering  hard- 
ware prototype  in  that  it  is  used  to  demonstrate  the  capabilities  of 
the  design,  is  used  to  furnish  data  for  iteration  of  changes,  and  is 
the  basis  for  the  user's  acceptance  of  the  work.  It  is  also  similar 
! * to  hardware  prototypes  in  that  no  cost  has  been  spared  in  making  it 

J 

functional,  but  will  be  changed  for  release  to  the  customer.  I fore- 

i • 

see  a large  simulation  facility  producing  such  prototypes,  which  would 

then  be  turned  over  to  a production  line  for  final  "assembly"  using 

i 

I • affordable  components. 

Can  we  establish  such  a facility?  What  are  the  required  ingre- 
I dients?  I see  the  need  for  a new  breed  of  software  artists,  equipped 

with  more  scientific  knowledge,  who  probably  will  have  to  be  the 
"chief  of  chiefs"  in  respect  to  the  programming  team  ideas.  He  must 
be  innovative,  must  have  a vast  support  facility,  both  in  hardware, 
but  especially  in  software,  and  must  be  allowed  to  operate  unfettered. 
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ile  would  be  {riven  much  leeway  in  the  selection  of  software  items  and 
design  tools,  lie  would  test  for  accurate  output,  but  would  not 
bother  testing  for  all  possible  outcomes  in  a given  programming  leg. 

He  would  write  in  a high  order  macro— lang\iage  and  would  not  concern 
himself  with  compiler  effectiveness.  The  finished  product  would  be  a 
bread-boai'd  on  which  the  user  would  either  place  his  stamp  of  approval 
or  request  changes  as  appropriate.  After  the  changes  have  been  made 
and  the  bread-hoard  approved,  it,  along  with  necessary  documentation, 
would  be  given  to  a final  design  and  programming  group  to  be  used  for 
preparing  the  final  design  specifications.  No  repair,  no  modification, 
no  "build-upon"  the  breadboard  would  occur.  The  breadboard  would  be 
thrown-away  after  it  is  no  longer  required. 

The  idea  of  the  breadboard  will  require  new  classes  of  tools. 

t 

But  the  nucleus  in  the  beginning  of  most  of  them  exist  now.  These 
tools  build  on  the  demonstrated  effectiveness  and  the  essentiality 
of  the  structured  design  and  programming  technology. 

The  process  outlined  above  for  the  production  of  the  software 
breadboard  would,  of  course,  not  be  applied  to  production  (i.e.. 


deliverable)  software  which  must  be  placed  in  maintenance.  Completed 
software  must  conform  to  standards  which  are  uniformly  enforced  in  the 
designer  community.  An  effort  amounting  to  13^  of  the  programing  just 
for  establishing  standards,  enforcing  them  and  verifying  the  code  is 
not  excessive.  Independent  third-party  testing  is  considered  essential 
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What  I have  discussed  so  far  are  my  views  on  developing 
software  in  the  face  of  realistic  user  requirements.  However,  we 
all  have  the  problem  of  "maintaining"  already  coded  software  systems. 
Maintenance  consists  mostly  of  changing  the  program  to  have  it  con- 
tinue to  meet  changes  in  the  user  requirements.  With  already- 
running  code  we  have  the  problem  of  identifying  parts  of  the  code 
which  may  have  to  be  altered  in  response  to  a new  user  requirement, 
or,  rarely,  in  response  to  a newly-discovered  logic  error,  and  in 
making  the  indicated  changes  without  introducing  errors  elsewhere 
in  the  program. 

The  first  need  is  to  be  able  to  read  the  existing  code.  While 
the  training  of  the  programmer  stresses  design  of  code  and  code 
writing,  in  fact  most  of  the  maintenance  programer's  time  is  spent 
in  reading  code.  Since  75^  of  our  effort  may  be  spent  in  program 
maintenance,  facility  in  code  reading  may  be  more  important  than  a 
capability  for  code  production.  Here  I borrow  from  Harland  Mills: 
The  training  we  give  our  programers  stresses  the  wrong  skills  too 
early.  An  analogy  from  English  literature  teachings  may  be  appro- 
priate: Only  after  the  student  has  achieved  a measure  of  mastery  of 
producing  critiques  of  someone  else's  work  is  he  allowed  to  produce 
his  own  creative  prose.  Harland  Mills  is  proposing  to  teach  pro- 
gramers to  read  code  first,  only  later  let  him  produce  code. 

To  help  the  reading  of  code,  it  should  be  put  into  a standard 
structural  form.  An  automated  flow  analysis  and  pattern  recognition 
can  help  in  producing  a structured  programing  form  as  was  shown  by 
the  restructuring  engine  concept.  While  there  is  no  doubt  that  the 
resulting  code,  produced  at  about  20%  of  the  cost  of  brand-new  code, 
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13  easier  to  read  and  may  be  cheaper  to  maintain,  this  represents 
but  a first  step  in  what  should  be  the  total  maintenance  activity. 

Most  of  our  large  business  codes  are  concerned  with  the  main- 
tenance, updating  and  cataloguing  of  very  large  files  and  data 
bases.  Some  of  the  more  efficient  methods  and  algorithms  for 
searching,  sorting  and  updating  are  but  three  to  five  years  old. 

Most  of  the  riinning  codes  are  older,  seven  to  ten  years  not  being 
too  atypical.  Hence  the  more  efficient  algorithms  are  not  in  the 
codes  being  maintained. 

While  machine  efficiency  is  not  a major  concern  with  most  com- 
puter operations,  many  of  us  are  faced  with  running  the  same  code 
on  say  one  hundred  machines.  In  this  case  any  small  savings  will  be 
greatly  multiplied;  but  even  in  cases  where  a given  application  may 
tend  to  saturate  the  computer  system,  changes  in  algorithms  can 
result  in  substantial  improvements. 

You  may  have  noted  that  nowhere  did  I refer  to  the  kind  of 
computer  we  would  use.  My  concern  is  with  meeting  the  user  require- 
ments first;  part  of  the  extended  simulation  I talked  about  earlier 
could  result  in  defining  a computing  structure  for  the  system.  To 
be  sure,  today  we  think  of  huge  machines  for  large  systems.  However, 
the  hardware  cost-trends  seem  to  indicate  that  it  is  not  unrealistic 
to  start  thinking  about  microprocessors  (or  small  single-bocurd 
computers)  that  have  the  processing  power  of  an  IBM  3^/30,  a 
machine  quite  adequate  for  many  business  applications. 

In  our  work  we  software  developers  must  continually  realize  that 
ultimately  we  serve  the  user;  and  in  meeting  our  customer's  require- 
ments we  must  ourselves  introduce  the  best  applicable  tools.  While 
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we  must  know  how  to  use  all  the  newest  tools,  we  must  also  know  when 
not  to  use  them.  I suggest  we  must  carry  the  latest  and  biggest 
sticks,  but  must  tread  very  softly. 

That  our  software  costs  are  killing  us,  is  no  secret.  Most 
likely  a relief  from  these  costs  will  come  from  a systematic  intro- 
duction of  more  computer  science  into  the  art  of  computer  programing. 
Without  question  it  is  necessary  to  be  creative  and  artful  in  the 
exercise  of  our  science,  but  we  are  getting  into  a mode  where  the 
bulk  of  the  programers  will  have  to  attend  to  the  day-to-day 
maintenance  of  codes,  at  least  during  part  of  their  working  life. 

We  must  therefore  devote  more  attention  to  producing  early  verifiable 
and  reliable  code  for  satisfying  users  and  more  maintainable  code 
for  easing  our  own  work. 

My  talk  this  evening  touched  on  a few  of  the  problems  and  visions 
for  their  solutions  that  have  surfaced  in  the  last  year.  Also  I had 
the  opportimity  to  get  to  know  many  of  the  leading  thinkers  in  the 
software. community,  to  help  set  a course  for  long-range  solutions 
and  to  dream  about  the  near- future  of  our  business.  Softwaire  engi- 
neering shows  an  exciting  future.  I foresee  effective  solutions  to 
our  current  dilemmas,  reachable  by  orderly  work,  by  further  exchange 
of  ideas,  for  the  germ  of  our  long-term  solutions  appear  to  exist 
already.  Symposia  like  this  one  mark  orderly  progress  in  our  quest 
and  point  to  further  fertile  developments. 

On  behalf  of  General  Hancock,  the  Army  Computer  Systems  Command 
and  ISRAD  I wish  you  a continued  successful  symposium. 
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SOFTWARE  MAINTENANCE  AND  LIFE  CYCLE  MANAGEMENT 


Clair  R.  Miller 
Honeywell  Information  Systems 

I recall  the  early  days  of  software  projects  where  the  assump- 
tion was  made  that  software  was  deficient  the  minute  you  produced 
it.  In  the  Uni vac  I days  the  constraint  of  a thousand  words  of 
memory  was  the  controlling  factor  of  the  entire  programming  process. 
It  was  obvious  then  that  software  had  a short  life  cycle  because 
everybody  wanted  something  better.  Phases  in  the  production  cycle 
like  design,  implementation,  testing,  and  maintenemce  were  tilted 
toward  the  front  end,  that  is,  the  designing  of  new  systems. 

For  business-oriented  applications  today,  it  is  estimated  that 
each  instruction  in  a computer  program  costs  $10  to  $15.  These  are 
the  same  figures  that  were  used  in  Univac  I days  so,  by  this  stamdard 
productivity  has  not  increased.  General  Slay,  ^ in  a recent  article 
in  Air  Force  Magazine,  quotes  the  cost  of  writing  an  average  FORTRAN 
instruction  at  $25.  He  states  that  the  Air  Force  spends  anywhere 
from  $100  to  $300  to  write  a software  line  of  instruction  for  com- 
plex, real-time  digital  systems!  For  the  SAGE  Air  Defense  System 
back  in  the  fifties,  the  cost  for  a line  of  instruction  was  estimated 
at  $50.  For  this  kind  of  programming,  productivity  may  have  actually 
decreased. 

Manufacturers  of  computers  are  vitally  interested  in  software 
productivity,  management  control,  and  resource  allocation  for  soft- 
ware projects.  If  you  look  at  the  balance  sheet  of  a computer 
manufacturer  it  is  heurd  to  find  the  eissets  devoted  to  softweire.  The 
corporate  level  allocations  deal  with  factories,  shipments,  emd 

^Slay,  Alton  D. , Lt.  Gen.  USAF,  "An  Air  Force  Avionics  Policy,"  Air 
Force  Magazine,  July  1977. 


i m 


BRKOXNO  PMM  BLAMC4IDT  rUMO) 


53 


revenues  tied  to  hardware.  However,  the  cost  of  developing  and  maintaining 
software  is  becoming  so  great  that  it  is  dictating  strategy  in  developing  new 
hardware  products. 

Computer  manufacturers'  software  is  the  interface  for  the  user  and  the 
hardware  and  must  meet  the  test  of  the  competitive  marketplace.  In  terminal- 
oriented  systems,  it  is  the  main  part  the  customer  sees.  This  is  somewhat 
different  from  applications  software  produced  for  one  particular  purpose  and  where 
accounting  procedures  are  not  a part  of  the  management  control  procedures. 

One  phase  of  software  production,  irrespective  of  whether  it  is  sold  or  used 
inhouse,  is  maintenance  of  software.  This  is  the  last  phase  in  the  life  cycle. 

The  resource  allocation  of  a software  project,  as  it  is  usually  conceived, 
is  depicted  in  Figure  1. 
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The  low  end  of  the  curve  is  the  maintenance  phase  which  supposedly  occurs 
after  'he  large-systems  design  and  programming  effort.  We  usually  plan  and  budget 
based  on  this  curve.  However,  what  usually  happens  is  a line  that  looks  like  Figure  2. 
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FIGURE  2 

If  inflation  is.  considered,  then  the  line  looks  like  Figure  3. 
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MITRE  Corp.,  in  a study  of  software  contracts  for  the  Air  Force  in  the 


Command  and  Control  area,  postulated  a picture  like  Figure  4. 
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They  found  that  large  Command  and  Control  projects  were  always  done  twice.  I 
have  a feeling  if  the  analysts  go  back  later,  they  will  find  they  are  in  the  third 
iteration. 

In  an  article  written  by  Rear  Admiral  Frank  S.  Haak,  U.  S.  Navy,  he  stated:^ 

At  the  present  time,  almost  two  out  of  every  three  in«>house 
progranners  and  computer  systems  analysts  are  involved  in 
maintaining  existing  data  systems. 

In  the  same  article,  he  further  stated  that  annual  costs  for  software  maintenance 
may  well  exceed  707.  of  the  initial  software  investment.  I believe  Barry  Boehm  of 
TRW  first  hit  the  707.  figure  in  October  1976. 

Recently,  while  I was  Investigating  a large  Air  Force  project  where  1200  programme'^s 
are  involved  in  replacing  old  systems,  I asked  what  the  progranners  would  do  if  all 
the  programming  for  the  new  systems  was  contracted  out.  The  answer  was  that  they 
would  be  busy  doing  maintenance. 

^Prokop,  Jan.,  Captain,  SC,  U.  S.  Navy.  Computers  in  the  Navy.  Annapolis,  Maryland:  ji 
Naval  Institute  Press,  1976.  j ^ 
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The  evidence  cited  fits  with  that  of  a picture  I believe  exists  for  computer 
vendors;  that  is,  the  major  portion  of  the  cost  of  software  over  the  Life  cycle  is 
in  maintenance.  Cost  data  is  scarce,  but  it  is  probably  safe  to  say  that  507.  to 
707.  of  total  costs  are  involved. 

If  the  major  portion  of  the  life  cycle  cost  is  maintenance,  then  there  is  more 
happening  than  just  error  correction.  What  is  happening  is  that  new  functionality 
and  instructions  are  being  added.  The  software  is  evolving  and  the  constraint  of 
maintaining  compatibility  between  the  original  design  and  the  new  additions  causes 
the  maintenance  task  to  stretch  out  over  a long  period  of  time.  Also,  a dominant 
force  now  is  change  in  architecture  of  systems.  There  is  a need  to  update  the 
hardware  piecemeal  which  means  you  have  old  and  new  hardware  within  one  overall 
system  architecture.  This  also  probably  means  old  and  new  software. 

Computer  manufacturers  are  well  aware  of  the  problem  of  maintaining  compatibility. 
When  new  machines  are  designed  they  carry  forward  the  instruction  set  of  old  machines 
so  that  old  programs  will  run  on  new  machines.  This  is  inefficient,  not  only  for  the 
old  programs  but  for  the  new  ones  as  well.  The  chances  are  more  instructions  than 
necessary  will  be  executed  in  the  new  programs.  Therefore,  after  a period  of  time , 
maintenance  of  programs  becomes  more  and  more  complex.  The  old  must  be  maintained 
as  well  as  the  new. 

In  studies  and  management  guides  devoted  to  software,  only  the  development 
process  is  considered.  There  is  an  assumption  that  once  it  is  developed,  tested 
and  declared  operational,  the  system  life  cycle  phases  and  major  milestones  are 
completed.  Theoretically,  the  resources  can  be  transferred  to  another  project. 


In  reality,  the  resources  are  kept  in  place  to  maintain  the  software,  and  new 
people  and  new  hardware  are  allocated  to  new  endeavors. 

Maintenance  of  software  is  a feedback  process.  The  key  to  good  feedback  is 
"Mass  Use  Overtime."  The  more  the  software  is  used,  the  better  it  gets  if  deficiencies 
are  fed  back  into  the  development  group  and  corrections  and  modifications  made. 

Also,  the  longer  it  is  used  the  less  the  probability  that  major  deflelencies  will 
occur. 

There  are  two  principles  that  should  help  maintainability.  In  the  initial 
design  it  must  be  clear  that  maintenance  will  have  to  deal  with  errors  that  are 
discovered  based  on  mass  use  overtime  and  extensions  and  modifications  of  the  software 
product  in  order  to  satisfy  new  requirements.  Therefore,  the  programs  must  have 
readability.  There  has  to  be  a deep  understanding  that  comes  from  reading  the  program;, 
over  and  over.  This  seems  to  be  one  of  the  main  justifications  for  structured  pro- 
gramming; that  it  simplifies  the  understanding  and  reading  of  programs.  The  structure 
has  built  in  consideration  for  downstream  support. 

The  second  principle  is  modularity.  The  modules  can  be  in  almost  any  form. 

Again,  this  is  a fairly  well  recognized  principle  of  structured  programming.  Break- 
points have  always  been  there,  but  now  more  emphasis  is  placed  on  making  it  easier 
for  true  modules  to  fit  together. 

These  are  principles  that  apply  to  the  completely  "new"  program.  They  are 
sometimes  hard  to  apply  to  a program  that  is  a mixture  of  "old"  and  "new."  This 
leads  back  to  the  basic  assumption:  programs  have  a life.  At  soma  point,  anything 
that  gets  rid  of  the  old  Is  good.  You  can  maintain  for  so  long,  and  then  the  package 
should  be  scrapped.  The  logic  that  is  used  is  that  even  if  the  new  is  bad  it  is  reallr 
good  because  it  gets  rid  of  the  old. 
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One  of  the  difficulties  in  maintaining  software  is  the  fact  that  the  process 
is  not  well  defined.  Equally  undefined  are  the  skills  that  are  involved.  People 
tend  to  specialize  in  certain  kinds  of  programs  such  as  compilers,  data  base  systems, 
operating  systems,  etc.,  but  nobody  wants  to  specialize  in  program  maintenance. 
Therefore,  the  tasks  involved  have  not  been  defined,  the  skills  involved  have  not 
been  taught,  and  exact  values  to  measure  accomplishment  is  not  available.  Programming 
is  essentially  a mental  discipline.  The  changes  in  this  mental  outlook  which  occur 
in  going  from  a design  phase  to  maintenance  phase  over  the  life  cycle  have  been  only 
partially  defined. 

There  are  a lot  of  behavioral  science  studies  that  lead  to  the  conclusion  that 
a group  of  5 to  10  people,  with  a leader  is  the  basic  organizational  structure  for 
a development  team.  I could  find  no  literature  that  applies  in  the  context  of  a 
team  for  "program  maintenance."  Generally,  it  is  understood  that  maintenance  is  one 
person  out  of  a central  location.  Some  kind  of  "bug  report"  is  generated  by  users. 
These  are  controlled  and  go  into  a data  base.  The  solution  to  a bug  is  not  always 
programming.  Sometimes,  there  is  functional  misuse,  documentation  faults,  and 
hardware  or  operator  faults.  When  a bug  does  involve  programming,  then  the  whole 
cycle,  including  test  and  evaluation,  is  triggered.  Thus,  each  bug  has  a life  cycle 
of  its  own.  Also,  bugs  frequently  not  only  affect  the  correct  operation  of  the  program 
in  which  the  bug  is  situated,  but  also  other  programs  or  system  operation  and  quite 
frequently  system  integrity.  In  a multiprogramming  environment  where  the  program 
executes  in  spurts,  the  beginnings  of  failure  may  have  occurred  a number  of  spurts 
before  the  one  in  which  the  crash  eventually  occurred. 

Programmers  can  spend  hours  and  days  tracking  down  a single  bug.  Usually,  this 
is  the  programmer  you  need  the  most  for  other  work.  If  the  pressure  is  heavy,  every- 
body can  get  weary  and  confused. 
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There  has  to  be  a family  of  tools  available.  The  operating  system  has  to 
record  recent  events , there  has  to  be  breakpoint  In  the  programs , and  there  has  to 
be  automatic  test  and  diagnostic  tools. 

In  DOD  software  management  studies,  the  solution  to  less  maintenance  is  perceived 
to  be  in  better  design  and  planning  initially.  The  structured  progranming  approach 
has  this  focus.  It  emphasizes  the  flow  of  information  through  the  program  and  the 
transformation  which  occurs  to  that  information.  Top-down  design  emphasizes  the 
entire  system  and  the  interrelationship  among  modules.  Thus,  the  whole  idea  is  to 
be  able  to  go  backward  and  see  what  is  happening./ 

The  technology  of  software  development,  which  focuses  on  how  you  do  it,  rather 
than  on  the  end  product,  is  changing.  It  does  not  appear  that  it  will  reduce  soft- 
ware production  time  or  affect  software  productivity,  but  it  does  appear  that  it  will 
produce  more  straightforward  software  and  thus  alleviate  the  maintenance  problem. 
However,  the  need  for  close  management  control,  planning,  and  attention  to  resource 
allocation  will  remain.  As  new  distributed  systems  software  becomes  operational, 
emphasis  on  maintenance  will  become  more  and  more  important.  Failures  will  be  much 
more  visible  because  systems  are  getting  bigger,  more  money  is  involved,  and  dis- 
tributed systems  spread  the  mistakes  over  a larger  geographic  area.  It  will  ba 
necessary  to  have  sophisticated  maintenance  people  who  have  all  the  resources  they  need 

There  has  been  progress  over  the  last  25  years.  It  has  been  evolutionary  rather 
than  revolutionary.  Things  happen,  not  only  because  they  are  technically  feasible 
but  also  because  of  the  dynamics  of  the  user-supplier  relationship.  Users  don't 
want  to  have  to  redo  everything.  They  want  a little  more  than  they  have  now.  This 
little  oiore  is  a maintenance  function  where  software  is  involved.  Such  an  incremental 
advance  does  not  come  out  of  universities  or  laboratories.  With  few  exception,  it 
comes  from  the  interaction  of  users  and  suppliers  and  from  small  groups  doing  soft- 
ware maintenance.  If  maintenance  is  the  way  you  progress,  then  resources  allocated 
to  this  function  should  receive  a higher  priority.  1 


SOFTWARE  MANAGEMENT  STANDARDS 


Dennis  W.  Fife 

Institute  for  Computer  Sciences  and  Technology 
National  Bureau  of  Standards 
Washington,  D.  C.  20234 


Today,  providing  computer  software  involves  greater  cost 
and  risk  than  providing  computer  equipment,  because  hardware 
is  mass  produced  by  industry  using  proven  technology,  while 
software  is  still  produced  mostly  by  the  craft  of  individual 
computer  programmers  and  users.  Over  70%  of  all  software  is 
developed  by  the  users  of  computers,  rather  than  the 
manufacturers  of  computer  equipment  or  the  producers  of 
commercial  software.  Software  is  very  expensive,  and  the 
development  coats  for  new  systems  are  growing  to  enormous 
levels.  The  Federal  government  is  estimated  to  spend  as  much 
as  $5  billion  annually  on  software,  and  its  accumulated 
investment  in  current  software  probably  exceeds  $25  billion. 
Software  costs  are  projected  to  become  9 times  as  great  as 
hardware  costa  by  1985. 

This  alarming  trend  has  several  causes.  One  of  course  is 
the  declining  cost  of  hardware,  which  gives  impetus  to  new 
system  concepts  and  expanding  applications.  Another  is  the 
growing  use  of  the  "software  approach"  to  hardware 
implementation  (i.e.  microprocessors  rather  than  wired  logic, 
to  provide  control  and  intelligence) . But  the  dominant 
reasons  are  the  labor-intensive,  often  undisciplined  character 
of  software  production,  and  the  lack  of  technical  foundations 
and  standard  practices  for  developing  reliable  and  valid 
software.  Delivered  software  typically  has  errors  at  a rate  of 
one  in  every  300  program  statements.  About  70%  of  software 
costs  are  currently  attributed  to  "maintenance",  which 
involves  correcting  latent  errors,  fixing  original  design 
flaws,  and  retrofitting  software  for  changed  requirements  that 
were  not  anticipated  in  the  original  software  development. 
Similarly,  nearly  50%  of  current  development  costs  typically 
are  consumed  in  software  testing  --  to  remove  errors,  to 
correct  poor  specifications,  and  to  fill  in  missing 
requirements.  Thus,  our  inability  to  control  software  costs 
mostly  stems  from  our  inability  to  control  the  design  process 
and  the  quality  of  the  end  product. 

Because  we  are  failing  so  badly  regarding  the  practical 
aspects  of  designing  and  producing  software,  "software 
engineering"  has  become  recognized  as  the  kind  of  discipline 
needed  to  evolve  solutions.  The  software  engineering 
movement,  now  about  three  years  old,  is  distilling  from  past 
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experience  the  beat  ideas  for  managing  software  projects  and 
exercising  technical  control  over  results.  This  paper 
discusses  the  content  and  approach  for  a potential  standards 
effort  that  would  assure  "best  practice",  considering  both  the 
management  aspects  and  the  programming  and  design  techniques 
for  software  projects  [1]. 


LIFE  CYCLE  MANAGEMENT 


Software  management  consists  of  all  the  technical  and 
management  activities,  decisions,  and  controls  that  are 
directly  required  to  purchase,  produce,  or  maintain  software 
throughout  the  useful  life  of  a computer  system  or  service. 

The  basic  framework  for  discussing  software  management  is 
the  life  cycle  viewpoint.  The  commonly  recognized  phases  in 
the  life  cycle  are  shown  in  Figure  1 , as  described  in  software 
documentation  standards  [2].  The  life  cycle  concept  is  not 
the  only  important  notion,  but  it  will  always  be  dominant 
because  it  is  a pragmatic  view.  After  all,  projects  are  paced 
by  higher  management  actions,  such  as  approvals,  budgeting, 
and  personnel  assignments,  which  occur  at  distinct  time 
points.  Also,  orderly  progress  on  a complex  undertaking 
requires  a step«wise  approach  toward  the  final  result,  with 
each  step  an  opportunity  for  evaluation  and  further 
preparation . 

The  INITIATION  phase  involves  the  assessment  of  an 
existing,  inadequate  system,  in  order  to  determine  the 
feasibility  of  replacing  it  with  an  improved  system.  The  four 
phases,  DEFINITION  through  TESTING,  are  the  development 
period,  where  a new  or  improved  software  product  is  being 
designed  and  implemented.  The  OPERATION  phase  starts  with  the 
operation  of  a new  system  by  its  intended  users,  and  includes 
the  subsequent  maintenance  and  minor  redesigns  that  user 
experience  may  dictate.  Eventually,  maintenance  no  longer 
suffices  for  needed  improvements,  and  another  development 
cycle  would  be  started.  The  following  summarizes  the  tasks 
included  in  each  phase. 


I 


Figure  1.  The  Life  Cycle  of  a Software  Product 


Time 


0 Major  technical  activity 
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In  the  INITIATION  phase,  software  management  assures  that 
the  technical  designers  understand  the  purpose  and  scope  of 
the  envisioned  system,  and  adequately  perceive  its  required 
functions.  The  INITIATION  phase  begins  when  management 
recognizes  that  a need  exists  for  a new  or  improved  computer 
service,  and  that  a study  should  be  made  to  formulate  a 
software  solution.  During  this  phase,  software  planners  or 
system  analysts  identify  the  intended  users  and  investigate 
their  work  processes.  The  planners  describe  the  services 
needed  from  the  envisioned  software,  and  then  conceive  and 
outline  alternativer  software  solutions.  A close  working 
relationship  with  the  intended  users  is  essential.  The 
alternative  approaches  should  be  described  in  terms  that  users 
can  understand.  User  concurrence  with  the  stated  requirements 
and  the  recommended  software  approach  should  be  a mandatory 
life  cycle  event. 

The  INITIATION  phase  also  assures  that  the  recommended 
software  approach  is  technically  and  economically  feasible. 
An  estimate  must  be  made  of  the  costs  and  benefits  for  each  of 
the  potential  software  solutions.  This  will  lead  to  a 
recommended  solution  and  will  provide  data  on  feasibility. 
Comparable  existing  systems  should  be  investigated  to  derive 
coat  data  and  to  confirm  that  proposed  solutions  are 
technically  realizable.  Comparisons  of  the  proposed  project 
to  past  failures  or  problem  areas  would  assure  that  these  can 
be  avoided  or  resolved  in  the  proposed  development.  The 
INITIATION  phase  concludes  with  a Feasibility  Report 
delineating  the  recommended  software  and  the  supporting 
analysis.  Management  may  then  approve  the  approach  and  budget 
the  estimated  funds  and  other  resources  for  acquisition  and 
operation  of  the  planned  system. 

The  DEFINITION  phase  defines  the  functional,  quality,  and 
performance  requirements,  that  are  needed  to  initiate  design 
as  well  as  to  confirm  that  the  software,  when  built,  will  meet 
its  objectives.  In  this  phase,  specifications  are  created  for 
the  functions  that  are  visible  to  the  users,  as  well  as 
predetermined  design  characteristics  that  are  crucial  to  the 
intended  usage.  In  other  words,  the  input  and  output  data, 
processing  operations,  and  design  constraints  are  defined. 
Performance  targets  are  stated  for  key  operations,  and  the 
design  attributes  for  software  quality  are  laid  down.  These 
Functional  Requirements  define  what  the  software  must  do  and 
the  general  aspects  of  how  it  will  be  constructed,  rather  than 
the  details.  The  specification  must  be  complete  and 
sufficiently  precise  for  outside  contracting,  if  desired. 
Equally  important,  the  specification  allows  the  intended  users 
to  confirm  that  the  proposed  system  will  meet  the  need  and 
will  have  acceptable  qualities  of  useability,  generality,  etc. 

The  DEFINITION  phase  also  provides  a Project  Plan  for 
managing  the  acquisition  and  installation  of  the  system.  The 
plan  includes  projections  of  the  detailed  costs.  A delivery 
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schedule  Is  laid  out,  Including  intermediate  milestones  and 
technical  reviews  for  evaluating  progress  and  insuring  that 
the  project  will  meet  its  goals.  Working  methods  are  defined 
for  quality  control  of  intermediate  and  final  products.  The 
plan  should  extend  to  the  installation  and  initial  operation 
of  the  system,  and  should  include  user  training,  document 
preparation  and  review,  acceptance  testing,  and  the  continuing 
obligations  of  any  contractors,  vendors,  or  support 
organizations.  The  Project  Plan  is  extremely  important 
because  it  specifies  how  the  effort  will  be  managed,  how 
problems  will  be  resolved,  and  how  the  quality  of  the  final 
system  will  be  assured. 

The  DESIGN  phase  produces  a thorough  Design  Specification 
for  guiding  a timely  and  problem>free  implementation  of  the 
proposed  software.  The  DESIGN  phase  begins  when  the 

Functional  Requirements  have  been  validated  by  intended  users, 
and  implementation  of  the  system  is  approved.  An  early 
objective  in  this  phase  is  to  complete  Design/Coding  Standards 
which  reflect  the  quality  requirements  stated  in  the 

DEFINITION  phase,  and  which  will  be  incumbent  on  every  program 
I produced.  The  DESIGN  phase  results  in  detailed  specifications 

! , of  the  Internal  construction  of  the  software,  for  use  by 

j programmers  who  will  implement  the  design.  This  involves 

I •.  definition  of  the  internal  architecture  of  the  software, 

performance  analyses  and  selection  of  basic  algorithms, 
delineation  of  individual  programs  or  modules,  specification 
of  interactions  between  components  of  software,  etc.  The 
Design  Specifications  and  Design/Coding  Standards  are  the 
means  for  resource  management  and  quality  control  during 

i PROGRAMMING. 


In  the  PROGRAMMING  phase,  software  management  assures 

i that  high  quality  workmanship  is  applied  in  producing  code 

**  meeting  the  Design  Specifications.  The  PROGRAMMING  phase 

often  is  merged  with  DESIGN  and  so  may  have  no  distinguishable 
r starting  point.  PROGRAMMING  involves  final  technical  choices 

1.  for  internal  program  design  and  the  creation  of  programming 

language  statements  or  "code*  that  is  executable  on  the 

r computer  and  that  implements  the  design.  Software  management 
( here  is  the  day-to-day  supervision  of  programmers,  guiding  and 
reviewing  their  tsobnloal  results.  PROGRAMMING  effort 

includes  the  testing  by  each  programmer  of  his  individual 

I programs  and  the  preparation  of  pertinent  management  reports 

and  software  documents.  Test  reports,  inspection  reports,  and 
program  documentation  sot  as  tangible  quality  controls  over 
I PROGRAMMING  aooomplishments . 


In  the  TESTING  phase,  software  management  assures  that  no 
significant  errors  or  design  disorepanoies  will  impede  usage 
of  the  system  in  the  intended  operational  environment. 
TESTING  proceeds  sooording  to  a Tost  Plan  prepared  in  the 
DESIGN  phase,  and  focuses  on  the  integration  of  individual 
programmer  results  in  an  overall  working  system.  Noted 
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discrepancies  and  errors  are  documented  for  correction  by  the 
programming  team.  Gross  shortcomings  In  the  original  design 
or  functional  specifications  may  be  exposed;  this  should  lead 
to  a major  project  review  In  order  to  plan  and  organize 
additional  definition  and  design  work. 

In  OPERATION  of  a delivered  system,  software  management 
assures  that  maintenance  Is  well  managed  and  controlled,  with 
no  less  effectiveness  than  the  original  development.  The 
OPERATION  phase  follows  the  delivery  and  Installation  of  the 
software  In  the  field,  and  Involves  periodic  redesign  and 
Improvement.  Software  management  procedures  that  are 
recommended  during  development  can  be  streamlined  to  provide 
similar  close  control  of  the  reduced  effort  committed  to 
corrective  design  work.  Thorough  specifications,  quality 
workmanship,  effective  testing,  and  management  review  remain 
highly  Important  In  directing  technical  effort  to  meet  cost 
and  schedule  targets. 

During  the  development  period,  the  early  tasks  may  be 
continued  or  repeated  In  order  to  remedy  design  defects  and 
Improve  quality  of  the  final  system.  Although  the  successive 
phases  defined  here  should  be  followed  whenever  practical,  the 
life  cycle  should  be  viewed  with  some  flexibility,  especially 
when  a project  Involves  unusual  complexity  or  Innovation.  For 
example.  It  may  be  Important  to  carry  only  a part  of  the 
system  through  to  DESIGN  and  PROGRAMMING,  In  order  to  confirm 
feasibility  before  beginning  DEFINITION  for  the  entire  system. 
So,  all  the  software  need  not  always  be  In  the  same  phase  of 
development . 

The  life  cycle  framework  establishes  a minimal  set  of 
standard  milestones  or  goals  for  managing  a software  project, 
whether  It  Is  a completely  new  development  or  a major 
upgrading  of  an  existing  system.  Figure  2 depicts  the 
milestones  that  are  of  moat  concern  to  customers  and  upper 
management.  In  addition  to  these,  the  project  manager  must 
formulate  Intermediate  milestones  during  the  periods  between 
these  major  events.  Project  milestones  must  be  concrete, 
measurable  Indicators  of  progress.  They  must  be  directly 
observable  by  the  project  manager  or  customer,  such  as  the 
receipt  of  a specified  document,  the  successful  operation  of  a 
given  program,  or  the  conoluslon  of  a scheduled  review 
meeting.  They  must  be  sufficiently  frequent  to  serve  as 
trouble  alerts,  and  to  give  enough  notice  for  action  before 
serious  repercussions  result.  Milestones  should  be  formulated 
to  permit  correcting  errors  or  omissions  In  specifications  and 
software  as  early  as  possible. 
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Figure  2.  Minimum  Milestones  for  Life  Cycle  Management 
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OPERAHOM 


PROGRAMING 


DESIGN 


DEFINITION 


INITIATION 


Description  of  Milestones 


Initiate  project  with  plan  for  feasibility  study. 

Complete  Feasibility  Report  with  recotnnended  software  solution. 
Complete  Functional  Requirements. 

Complete  Project  Plan. 

Complete  Test  Plan;  and  Design/Coding  Standards. 

Complete  Design  Specification  of  all  component  programs. 

Complete  unit  testing;  accept  all  program  source  code  for  Integration 
testing. 

Initiate  fault  reporting  procedures. 

Complete  Integration  testing  according  to  Test  Plan. 

Complete  delivery' and  shakedown. 


Each  milestone  In  Figure  2 should  be  an  occasion  for  an 
intensive  review  that  covers  both  technical  and  management 
issues.  Participants  would  include  higher  management  and 
customer  representatives,  but  of  course  the  key  participant  is 
the  project  manager.  At  each  review,  the  project  manager 
would  present  the  current  project  status,  giving  expended 
versus  allocated  resources,  any  resource  problems  or  technical 
concerns,  and  pertinent  details  of  his  planning  to  accomplish 
subsequent  milestones.  The  deliverable  product  under  review 
would  be  outlined  briefly  before  commencing  detailed  comments 
and  analysis.  The  reviewers  should  include  technically 
knowledgeable  people  who  are  able  to  vouch  for  the  adequacy  of 
the  product.  Any  discrepancies  or  needed  improvements  should 
be  documented  in  a report,  and  accepted  by  the  project  manager 
for  immediate  rework  and  resolution.  The  milestone  should  not 
be  considered  complete  until  the  modifications  have  been  made 
or  else  follow-on  milestones  have  been  approved  as  changes  to 
the  Project  Plan. 

The  process  of  periodic  reviews  is  aimed  at  validating 
specifications  and  software  results  for  responsiveness  to  the 
customer's  need  and  consistency  with  the  project  resources. 
Each  review  presents  management  with  the  need  to  reassess  the 
productivity  of  the  project  team  and  the  resources  needed  for 
meeting  the  remaining  project  goals. 

Continuing  management  involves  both  resource  managment 
and  technical  control.  If  problems  are  evident  in  meeting 
cost  and  schedule  objectives,  management  may  seek  additional 
resources  or  make  better  use  of  available  resources.  Or, 
management  can  decide  tc  reduce  the  product  quality,  and  act 
to  deliver  a poorer  or  lesser  system  within  the  existing  cost 
and  schedule  targets.  The  latter  course  often  may  be  taken, 
because  higher  management  and  customers  usually  cannot 
perceive  this  happening,  and  it  postpones  the  potential  day  of 
reckoning . 

The  underlying  source  of  software  problems  has  been 
stated  in  various  ways.  Some  authorities  call  it  overambition, 
that  is,  attempting  to  build  systems  that  are  beyond  our 
experience.  Poor  productivity  is  also  claimed  as  the  chief 
culprit.  Both  of  these  factors  are  certaihly  troublesome,  but 
the  more  fundamental  problem  is  probably  poor  quality  control. 
To  build  systems  on  time  and  within  cost,  we  must  understand 
the  relationship  between  project  cost  and  resulting  product 
quality,  as  well  as  the  process  of  controlling  quality  as 
development  or  system  operation  proceeds.  Yet,  there  are  no 
commonly  understood  definitions  of  quality  factors  for 
software,  and  there  is  little  reason  to  believe  that  the 
proven  practices  of  quality  software  production  are  being 
widely  adopted.  A major  goal  in  life  cycle  management  should 
be  to  establish  appropriate  standards  of  practice  that  will 
provide  quality  assurance  as  a routine  activity. 
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THE  PURPOSE  AND  NATURE  OF  STANDARDS 


It  l3  custonary  in  coksputer  standards  to  think  in  terms 
of  a product  standard,  that  is,  a specification  for  some 
hardware  or  software  item  that  may  be  purchased  or  built.  In 
the  general  field  of  standards,  however,  considerable  effort 
is.  spent  in  developing  standards  of  practice,  that  is,  methods 
and  criteria  for  accomplishing  tasks  or  analyses  that  are 
necessary  to  the  engineering  or  construction  of  desired 
products.  The  only  software  standards  that  are  close  to  being 
standards  of  practice  are  the  few  on  documentation,  which  are 
mostly  concerned  with  superficial  mechanics,  such  as  the 
layout  of  coding  forms.  Clearly,  this  is  a poor  situation,  if 
you  accept  the  idea  that  software  design  and  development  is 
not  a routine  task  that  is  quickly  learned  and  well  performed 
by  any  person  of  average  intelligence. 

Standards  of  practice  for  software  management  and  quality 
assurance  could  accomplish  a number  of  important  purposes. 
First,  software  managers  need  standards  that  would  form  an 
authoritative  basis  for  estimating  the  minimum  resources  and 
schedule  required  for  a proposed  software  project.  In  other 
words,  standards  could  help  resolve  feasibility  questions,  and 
would  prevent  ill-advised  customer  demands  for  highest  quality 
even  though  funds  and  schedules  were  being  reduced  below  the 
feasible  level. 

Second,  standards  of  practice  would  guide  the  new 
software  manager  who  enters  on  this  responsibility  from 
another  field,  without  adequate  prior  experience.  Despite 
advances  in  professional  education,  many  people  still  seem  to 
be  thrust  into  decision-making  roles  on  projects,  having  only 
a general  engineering  or  technical  background,  or  routine 
programming  experience. 

Third,  standards  are  necessary  to  instill  an  engineering 
approach  and  modern  techniques  into  all  projects.  The  rapid 
evolution  cf  software  technology,  in  leas  than  a generation, 
presents  us  with  too  many  people  who  have  limited  skills  or 
outdated  work  approaches  that  cannot  meet  the  quality 
requirements  for  modern  complex  systems. 

On  the  other  band,  standards  of  practice  cannot  be  a 
substitute  for  adequate  and  oontinulng  training.  Standards 
should  prescribe  basio  methods  or  the  essential  elements  of 
techniques  that  are  recommended,  in  a way  that  achieves 
simplicity  of  use  and  ready  understanding  for  the  technically 
oriented.  Thorough  understanding  and  fluent  use  would  have  to 
come  through  supplemental  training  aimed  at  establishing 
consistent  use  of  the  standard.  The  standard  serves  then  as  a 
control  and  a guidepost  for  everyday  practice. 
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THE  QUALITIES  OF  SUPERIOR  SOFTWARE 


Software  fflanagement  may  be  used  to  control  quality  at  an 
acceptably  high  level  while  conserving  the  time  and  resources 
needed  for  a project,  or  It  may  be  used  to  maximize  quality 
lasofar  as  possible  with  the  resources  and  time  available. 
Quality  la  software  la  a complex  Issue.  Nevertheless,  there 
are  general  properties  that  are  characteristic  of  high  quality 
software : 

CORRECTNESS— Programs  perform  exactly  and  correctly  all 
the  functions  defined  In  the  specifications.  If  available,  or 
else  In  the  documentation.  This  means  that  Incorrect 
documentation  Is  as  serious  as  an  Incorrect  program. 
Correctness  is  an  Ideal  quality,  that  Is  rarely  determinable, 
so  a more  practical  quality  Is  RELIABILITY. 

RELIABILITY--Programs  perform  without  significant 
detectable  errors  all  the  functions  defined  In  the 
specifications  or  the  documentation.  High  reliability 
Indicates  that  programs  are  relatively  trouble  free  In 
performing  what  they  are  claimed  to  do.  But  an  equally 
important  question  Is  whether  the  functions  and  performance 
are  adequate  and  suitable  to  a needed  purpose.  The  latter 
quality  is  called  VALIDITY. 

VALIDITY— Programs  provide  the  performance,  all 
functions,  and  appropriate  Interfaces  to  other  software 
components,  that  are  sufficient  for  beneficial  application  In 
the  Intended  user  environment.  This  means  the  software, 
without  additional  programming  or  manual  Intervention,  has  the 
capabilities  to  do  the  Job  It  Is  supposed  to  do.  Validity  Is 
a quality  of  specifications  as  well  as  computer  programs. 
Examples  of  an  Invalid  program  would  be  an  interactive  editor 
that  had  no  online  function  for  retrieving  stored  text  for 
Inspection,  or  a FORTRAN  language  compiler  that  had  no  DO  loop 
Implementation.  Validity  involves  Judgement  of  user 
requirements,  and  may  change  If  the  Intended  application  or 
purpose  Is  altered.  Because  poor  reliability  may  render  a 
needed  function  useless,  reliability  Is  necessary  to  validity. 

RESILIENCE  (or  ROBUSTNESS)— Programs  continue  to  perform 
in  a reasonable  way  despite  violations  of  the  assumed  Input 
and  usage  conventions.  Input  of  unacceptable  data,  or  an 
Inconsistent  command,  should  never  cause  a result  that  Is 
astonishing  and  detrimental  to  the  uaer--such  as  the  deletion 
of  any  valid  results  obtained  previously.  Programs  should 
Include  routine  checks  and  reccvery  possibilities  that  are 
"forgiving”  of  common  user  and  data  errors.  Resilience  Is 
related  to  the  broader  quality  of  USEABILITY. 

USEABILITY— Programs  have  functions  and  usage  techniques 
that  are  natural  and  convenient  for  people,  showing  good 
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consideration  of  human  factors  and  limitations.  For  example, 
useable  programs  have  few  arbitrary  codes  for  data  In  Input  or 
output,  have  consistent  conventions  In  different  operating 
modes,  and  provide  thorough  diagnostic  messages  for  errors  or 
violations  of  use. 

CLARITY--The  functions  and  operation  of  the  programs  are 
easily  understood  from  the  user  manual,  and  the  program  design 
and  structure  are  readily  apparent  from  the  listing  of  program 
statements.  This  means  that  documentation  must  be  well 
written,  but  also  that  the  program  Is  carefully  designed,  with 
meaningful  choices  of  variable  names,  use  of  known  algorithms, 
frequent  and  effective  comments  In  the  program  to  describe  Its 
operation,  and  a modular  structure  that  Isolates  separate 
functions  for  examination. 

MAINTAINABILITY'-Programs  are  well  documented  by  manuals 
and  Internal  comments,  and  so  well  structured  that  another 
programmer  could  easily  repair  defects  or  make  minor 
Improvements.  Clarity  Is  essential  for  maintainability.  But 
maintainability  also  Implies  a wide  variety  of  good  design 
attributes,  such  as  program  functions  that  help  to  diagnose 
potential  problems,  e.  g.,  periodic  reports  of  status  or 
control  totals;  or,  general  techniques  that  readily  support 
changes,  e.  g.,  the  Isolation  of  constants,  report  titles,  and 
other  static  data  as  symbolic  Items. 

MODIFIABILITY— Program  functions  that  might  require  major 
change  are  well  documented  and  Isolated  In  distinct  modules. 
Maintainability  Is  essential  to  modifiability.  But 
modifiability  means  that  a concerted  effort  was  made  to 
anticipate  major  changes,  and  to  plan  the  software  design  so 
that  they  could  be  made  easily. 

GSNERALITY-'-Programs  perform  their  functions  over  a wide 
range  of  Input  values  and  usage  modes.  Programs  are  not 
limited  to  special  cases  or  ranges  of  values,  when  the 
functions  are  commonly  or  reasonably  extendable  to  a more 
general  case. 

PORTABILITY— Programs  are  easily  Installed  on  another 
computer  or  under  another  operating  system  program.  Standard 
programming  language  Is  used,  and  hardware  or  other  software 
dependent  features  are  Isolated  for  easy  change. 

TESTABILITY— Programs  are  simply  structured  and  use 
general  algorithms,  to  facilitate  step-by-step  testing  of  all 
oapabllltlos . 

EFFICIENCY/ECONOMX— Programs  have  high  performance 
algorithms  and  conservatively  use  computer  resources,  such  as 
main  storage,  so  that  the  cost  of  program  operation  Is  low. 
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DESIGN  STANDARDS 


The  above  quality  definitions  should  be  prominent  in  the 
mind  of  every  customer  and  software  planner  during  the 
DEFINITION  and  DESIGN  phases  when  specifications  and  design 
s.tandards  are  being  developed.  Although  the  above  are  general 
statements,  they  are  a starting  point  for  evolving  a set  of 
uniform  design  standards  that  would  serve  as  a foundation  for 
meeting  quality  requirements.  An  approach  la  to  refine  the 
definitions,  separating  different  aspects  of  each  criterion, 
and  then  proceed  to  define  specific  features  that  would 
represent  the  achievement  of  each  criterion  when  implemented 
in  every  program.  Of  course,  there  also  may  be  special 
requirements  that  only  pertain  to  one  system  function  or 
program.  For  example,  maintainability  has  two 
aapects>-capablllty  to  diagnose  errors  or  problems,  and 
capability  to  change  programs.  A general  software  feature 
that  supports  maintainability  in  regard  to  changing  programs, 
is  the  use  of  only  standard,  high  level  programming  language. 
A special  requirement  that  may  be  needed  for 
economy/efficiency  might,  for  instance,  be  to  implement  some 
particular  table  search  or  selection  algorithm  in  assembly 
language,  rather  than  the  high  level  language  used  for  all 
other  modules. 

The  result  of  pursuing  quality  requirements  in  this  way 
will  be  a design  standard  in  the  form  of  a checiclist  that  can 
be  used  in  team  reviews  or  inspections,  and  in  many  cases,  in 
automated  code  auditing.  Although  the  need  for  such  a design 
checklist  seems  obvious  and  long-standing,  one  hasn't  been 
published  yet.  Programming  textbooks  sometimes  have 
suggestions  of  this  type  scattered  among  the  example 
programming  exercises.  The  problems  of  software  quality  only 
recently  have  spawned  a few  books  and  papers  that  collect  such 
"design  wisdom”  for  programmers  [6-9]. 

Program  design  standards  must  be  sufficiently  precise  to 
be  objectively  measured  on  an  individual  basis.  That  is, 
satisfaction  of  each  standard  must  be  either  automatically 
determinable  by  a program  analyzer,  directly  observable  by 
inspecting  the  code,  or  demonstrable  or  testable  by  running 
the  program.  This  is  the  only  way  that  a basis  can  be 
established  for  designing  in  quality  and  for  measuring  quality 
during  development  and  at  delivery. 


QOALITT  CONTROL  STANDARDS 


A variety  of  quality  control  techniques  have  been 
practiced  in  software  development,  with  some  benefit.  They 
are  not  consistently  used.  Sometimes  there  is  an  adequate 
reason  for  not  using  one  or  another,  because  the  cost  in 
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personnel  time  and  effort  Is  greater  than  the  expected 
benefit.  But,  considerable  effort  is  being  made  at  this 
moment  on  guidebooks  or  standards  to  formalize  the  practice  of 
some  techniques  in  many  organizations  [1-5].  In  addition  to 
these  references,  NASA  is  preparing  software  management 
guidelines,  and  the  IEEE  Computer  Society  Technical  Committee 
on  Software  Engineering  is  working  to  draft  a standard  for 
industry.  Figure  3 is  a concise  listing  of  many  of  the 
accepted  quality  controls.  It  is  clear  that  most  of  them  are 
not  automated  or  purely  technical  methods,  but  rather  are 
judgemental  methods  relying  on  documentation. 

Narrative  documentation,  rather  than  program  statements 
or  other  symbolic  codes,  is  the  primary  basis  for  quality 
control  of  software  at  this  time,  for  several  reasons: 

1)  Almost  any  degree  of  quality  is  possible  for  software; 
what  degree  can  be  afforded  must  be  agreed  through  the 
project  documentation. 

2}  No  better  means  is  now  accepted  for  specifications  than 
narrative  documents. 


3)  Software  is  a "black  box"  to  users,  and  documentation  is 
needed  to  test  and  exercise  it. 

Because  of  this,  quality  control  standards  to  a large  extent 
must  address  the  format,  content,  and  means  of  preparing  and 
updating  documents.  Computer  software  for  text  editing, 
filing,  and  document  formatting  are  important  tools  for 
quality  control. 


STANDARD  SOFTWARE  PRODUCTION  TOOLS 
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Software  production  urgently  needs  more  use  of  automated 
tools  and  less  use  of  judgemental  methods  for  quality  cont.''ol 
and  implementation  of  program  code.  As  shown  in  a recent 
analysis  of  projects  [10],  the  factors  with  greatest  impact  on 
productivity  are  these  involving  project  complexity,  personnel 
experience,  and  customer  interaction,  where  there  is  little 
possibility  of  technological  help.  So  attention  must  turn  to 
the  routine  technical  aspects  of  software  development  that 
arise  in  the  DESIGN,  PROGRAMMING,  and  TESTING  phases.  High 
level  programming  languages  and  interactive  programming 
support  have  proven  that  major  improvements  occur  in 
programmer  productivity  and  software  reliability,  through 
better  software  for  programmer  support.  The  trend  to  improve 
programmer  tools,  that  started  with  these  advances  over  a 
decade  ago,  needs  to  be  reinforced.  The  evolution  of 
structured  programming  has  pointed  out  the  value  of  program 
librarian  and  source  code  control  systems,  as  well  as 
innovations  in  programming  language  features.  Source  code 
analyzers,  even  though  often  used  today,  need  further 
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advancement  toward  a capability  of  auditing  for  conformance  to 
design  and  programming  stardarda.  Basic  research  is  needed  on 
software  testing  aids  of  various  types,  such  as  automated  test 
I data  generation,  test  evaluation  and  control,  and  fault 

f diagnosis,  particularly  to  evaluate  their  benefits  in 

I detecting  various  types  of  program  errors  or  design  flaws. 

I Despite  the  fact  that  software  aids  to  programming  are 

[ nothing  new,  surveys  [11-12]  indicate  that  a good  repertoire 

of  tools  is  seldom  available  or  consistently  used.  A 
standardization  effort  is  needed  to  stimulate  availability  of 
uniform,  low-cost  tools.  Since  the  design  of  a specific  tool 
may  be  highly  dependent  on  the  language  of  target  programs  or 
on  the  host  operating  system  software,  the  appropriate 
approach  would  probably  concentrate  on  functional 
specification  of  generic  types  of  tools  and  their  application, 
rather  than  a product  standard  for  one  universal  tool  of  each 
kind . 


Figure  3*  A List  of  Quality  Controls 


PROJECT  RECORD — Official  Journal  of  plans  and  decisions. 

DOCUMENTATION  STANDARDS-->Rules  on  scope,  content,  and  format. 

STRUCTURED  SPECIFICATIONS— Format  coordination  for 

traceability  from  requirements  through  testing. 

QUALITY  REQUIREMENTS— Separate  specification  of  quality  design 
criteria . 

UNIT  FOLDERS--Standard  programmer  records  of  module  status  and 
design  In  progress. 

FAULT  REPORTS--Reports  of  observed  software  errors  In 

operation . 

SPECIFICATION  CONTROL--Control  of  proposed  changes  to 
specifications . 

SOURCE  CODE  CONTROL— Control  of  changes  to  delivered  program 
modules . 

TESTING  STANDARDS — Uniform  criteria  for  all  testing. 

INDEPENDENT  TEST  PLANS->-Independently  prepared  tests  of 

Integrated  software. 

CONFIGURATION  MANAGEMENT— Control  of  overall  system  status  and 
releases . 

TEAM  REVIEWS— Conferences  to  audit  design  and  completed 
programs . 

INSPECTIONS--Independent  audit  of  program  subsystems. 

PROJECT  INFORMATION--Computer  Information  support  to  resource 
management . 

PROGRAM  ANALYZERS— Soft  ware  to  audit  source  code. 

PRODUCTION  TOOLS— Compiler , editor,  debugger,  etc. 

STRUCTURED  PROGRAMMING--Improved  design  and  programming 
technique . 

TOP-DOWN  DEVELOPMENT--Bulldlng  programs  starting  from  the 
application-oriented  level. 
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CONCLUSIONS  AND  RESEARCH  IMPLICATIONS 


There  Is  no  evidence  or  reason  to  believe  that  software 
development  cannot  be  managed  through  many  of  the  commonly 
recognized  approaches  to  project  management  and  control.  Of 
course,  software  terminology  must  be  brought  In,  and  there  are 
special  milestones  to  be  defined,  but  much  of  the  frameworic  of 
system  life  cycle  management  Is  applicable  and  useful.  The 
long  overdue  step  Is  to  Insure  that  It  Is  used  by  every 

organization  In  every  project,  despite  the  modest  extra  effort 

required.  Many  government  agencies  are  Initiating  standards 
or  guidelines  as  directives  on  future  projects. 

But,  regulation  In  terms  of  management  procedures  or 
project  organization  Is  not  enough.  Standards  of  practice  are 
needed  that  lay  down  the  technical  methods  and  criteria  that 
should  be  followed.  Establishing  appropriate  standards  Is  a 
significant  research  effort.  A great  deal  of  technical  wisdom 
must  be  assembled,  evaluated,  and  codified  In  understandable 
English  and  mathematics.  Standards  must  be  scaled  to  project 
resources  and  the  life  cycle  objectives  of  a product.  This 
means  that  the  evaluation  effort  may  be  a considerable 

challenge,  to  determine  properly  the  applicability  of 

prospective  methods  to  different  projects. 

Near  future  breakthroughs  In  such  an  approach  are  likely 
to  concentrate  on  design  and  programming  tools  that  automate 
many  of  the  routine  tasks  of  software  development.  Quality 
control  Is  the  area  needing  work,  particularly  toward  auditing 
and  validation  of  programs,  and  the  attainment  of  thorough 
testing . 

The  area  least  subject  to  routine  Improvement  Is  one 
which  will  still  benefit  from  the  human  skills  freed  up  by 
Improvements  elsewhere.  This  means  design,  and  particularly 
the  creative  process  by  which  a skilled  person  turns 
unstructured  functional  requirements  Into  a feasible  software 
concept,  that  can  be  Implemented  with  a given  amount  of  time 
and  money.  Significant  help  here  could  still  come  from 
standards  that  present  model  system  designs  as  cost  and 
I quality  benchmarks,  or  that  offer  nominal  resource  estimation 

, procedures  to  guide  management  decisions  on  project 

[ Investments  and  quality  requirements. 
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so^tn*r»  «i9in*«rfnq 
issupanct 
sofr>*r»  suality 

softMr*  DMsurarant  and  «v«iiMClan 

duality  iMtrtcs 

diidllcy  cl«rtct«rf sties 

TonadtiMflC  ty  eojectivea 

soi’tMn  ftanoards 

softMra  rtPatnity 

tasting 

Abstract 

TTw  study  Pdsorttd  in  this  saoar  astabllsnas  a 
caneaptual  ^rantMrt  and  som  <ay  Initial  rasults  In 
tJta  analysis  sue  cJiaractarfstlcs  sf  soi'twar*  dual- 
ity. Its  3ia1n  '■asults  and  csncluslons  art: 

» Explicit  attanti on  to  cHarictarf sties  af  soft- 
imra- duality  can  laad  to  significant  savings 
In  softMrt  llfa-cyclt  costs. 

» Tlia  currant  softxara  stata-of-tfta-art  lnwosas 
spacific  limitations  an  our  anility  to  auto- 
aatleal'y  and  duantitacivaly  avaluata  tn» 
duality  of  softwara. 

a A dafinitiva  nlararcny  af  xall-dafinad,  v«all- 
dlffarantiatao  cnaractari sties  of  saftMara 
duality  Is  aavalooad.  la  filgnar-laval  strjc- 
tura  rafToca  t.ia  actual  jsas  to  «nicn  soft* 
aura  duality  svaluatlon  i^uld  aa  aue;  its 
lowar  laval  tnanctaristics  ira  clcsaly  tarra- 
latad  wltn  actual  softwara  ‘natnc  tvaluaclons 
Hdleti  can  aa  aarfarmad. 

a A larga  mmar  of  softwara  auallty-^aluacion 
aatrfcs  Itava  baan  daflnad,  ciassiflau,  and 
avaluatad  wltt  rasoaet  to  tnair  aotantial  aana- 
f1ts«  duantifiaoil  Ity,  and  tasa  of  automation. 

» Particular  softwara  llfa-cycla  activltlas  Hava 
baan  Idanttflad  wnicfi  bava  significant  lavar* 
aga  on  softwara  Quality. 

iBoortantly.  «a  baliava  tnat  tna  study  ra- 
portad  In  this  oacar  arovieas  for  t-ta  fl>^t  tira  a 
elaar.  wall-daf1nad  framtwort  for  satass’ng  tta  oftan 
slippary  issuas  sssoclatad  witn  seftivart  aiiaHty,  via 
ttia  eonslstant  and  lutuaily  suooortfva  sats  af  tafim- 
elons,  distinctions,  guloallnas.  and  ncar'ancis 
citad.  This  frsaawort  is  cartairlv  ;iot  ttre'att,  but 
It  has  baan  brougnt  to  a aoint  saffidant  to  sarva  as 
a vlabla  basis  for  futura  raflnamants  ana  txtanslons. 


1.  Introduction 

^ Evaluata  Softwara  duality?  Supoosa  /ou  racalva 
a softwara  oroeuct  -.nitn  's  tei'varsd  an  :*~t.  -<>tnin 
budgat,  and  <«n'c.h  corrsctl/  ino  *f"c*.tnti..  .'•'■■'cr-s  ail 
Its  spacifltd  functltns.  leas  it  'ol'cw  "at  /ou  ■•'"I 
ba- happy  wiui  it?  "or  lavaral  '•aasons.  zr.t  s/is~er  -ay 
be  "no."  Hora  ira  scca  af  tna  conmin  aroolams  you  ray 
find: 

1.  Tha  sofyra  aroduct  nav  ;a  tard  *s  .jndarstsnd 
andfll^ficu.:  ta  -ooifv.  “^i^s  tscs  to  sxcas- 
slva  costs  'n  soi't.vara  ’•iintenanca.  ana  t-tsa 
costs  ara  not  trlv^I . .-or  sxamoia.  a rtcart 
saoar  by  Etsnoff-l-l  indicatts  that  75  agrtsnt 
af  3anara!  'Motors'  softwara  affort  is  scant  in 
softwara  nalntenanca,  and  ciat  is  'a1  r'y 
tyoicai  of  larga  industry  softwara  activitits. 

2.  Tha  softwara  aroduct  nav  aa  -i1f**tult  to . ,§£. 
tr  tasv  to  ni-usa.  ■«  ■■‘scant  jAC  ••scorf-- 
^oantltlad  avar  ';0. COO, COO  in  ■uinaeassarv 
Sovanwitnt  casts  aua  to  ^OP  aroolars:  nany  af 
than  wara  bacausa  tna  softwara  <»as  so  tasy  to 
alsusa. 

3.  Tha  softwara  crcouct  nav  ba  unnactssarUy 

aachina-cacareent.  ar  -are  to  -ntaertta  *it.h 
othar  oroersrs . ~^i5  aroeign  !s  a:*f<cu!t 

anougn  now.  out  is  nacnina  tyces  tontinua  to 
aroHfarsta.  it  mil  ;at  worst  sno  worsa. 

’*a.ior  Softwara  "ualltv  Taclslon  a^lnts.  *hara  sra 
1 lunar  of  ‘amiiMr  situations  n <»n:c,n  t 's  rcssibla 
to  axart  a strong  'aruanca  an  saft-ara  tual'ty.  snd 
"or  •nicn  It  is  ■moortinc  M lava  i good  unoarstsnoirg 
af  aia  various  cnaractartsTics  af  softwara  auailty. 

Hart  »ra-  < fsw: 

1.  Praoarlno  tha  auallrv  so^ifiestlcns  for  s 
softwara  arteuct.  .-armulsclnd  wnat  "'inctians 
you  naao  ano  *ow  nuc.t  oar*om«anct  ' scaad,  ic- 
airaey)  you  naad  ara  fair’/  smlcntforwaro. 
Indicating  that  you  also  nood  mairtalnaelllty 
or  undorstanoaoHlty  Is  imeortant,  out  much 
mort  difficult  to  fonmilact  In  soma  tastaolt 
fashion. 

2.  Thaclclno  far  cawol’anea  with  eualltv  socc-*1ca» 
•Ions,  rhts  ‘s  assontiai  tna  tuai’tv  scad- 
^Tcaclans  ara  to  aa  laanlngf'jl . It  can  alaarl/ 
oa  dona  win  a larga  Hvasetant  af  good  aaoola. 
aut  this  sert  af  cnacking  ’s  toih  axoansiva 
and  hard  an  aaoola's  nrala. 


•Previously  published  in  the  Proceedings  of  the  Second  International  Con- 
ference on  Software  Engineering,  San  Francisco,  October  19^6. 
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I.  mtitw  c-  tndtofft  bttw—t  davl- 

wiMiicaiM  wo  octr»siaR<i  cas«7^:*<is  is 
•spaclally  iarartuit  bacaus*  clqnr  davcIooMnc 
budvtts.  ar  scntdulM  caus»  arajtets  ta  iklnt 
oir  ■tlnta.fiublllty.  oorubillty,  tnd 
uMbtTICy. 

♦.  Saftwarr  B«ck«a»  s»<acrtoir.  Har»*9a1a.  laany 
man,  4 rat4c;y»  usassiaiic  af  ha»  aasi  1y 
aadr  packt9»  cair  a»  abaocad  ts  uialr  <nsttn*> 
tlon'i  etiaii9ln9  naaas  and  ftarMr*-  bas«t. 

TYirprtaary  payoff  a^atr  ineraasad  eaoabITtty  ta 
daal  «<tlr  MftMr*  tuaUty  cansidaratioas  Mwld  bdair 
lapiwMaane  la  tof'9>ar«  •waintraiiaiicar  caat-affaet1vaiiaa«. 
To*  Httldof  a auaUcy  (•.!.»  aalntoliiaai  11  ty)  traiW' 
latw  dtracely  inw  ravaar  eoar  (1.»..  cost  of 
11f*-<yel»m1i»taaf  c»  luea  u earraerloir  of  arrwr* 
aad  roraoaa*  ta  rmr  ^isar*  rmu]  r '«aBtx) . I<r  v1a»  of 
Ohs  slapi*  traor.  t Is  rtoiar  > uvrislnd  tftac  oar*' 
sarlous  and  dafinir;  *■01*  had  noe  baao  aea*  ta  data 
ti»  tlwtrdaof  avaluar.nd  taftMr*4«iaI1ty. 


avaluac^^^twan*^!  1 ty  leaaart  ta  hav*  first  haM- 
attvdtad  lir  ur  ardaflltadroay  ay  ^uboy  anr  Hartw1efc.^^< 
Til*  oatliod  of  Haf . 1 .<as  ta  tofin*  coo*  ‘'attrtbotas'* 
and  Oiadr  ’oatrfcr^"  ai*  fanrar  oalnf  a pros*  aaoras- 
sim  9f  ttm  oardealar  quality  oaalrad  of  tn*  softMrov 
aad  tb*  lattar  a satbanticai  ^uictlair  of  oanaMtars 
dwidbc  ta  roiat*  ta  or*  dafin*  oi*  attribut*.  Attn* 
bataa  sudr  as:  'St  - satnamtlcai  calculations  ar* 
eorractly  parforoad;.*  ar  *Af  - Tli*  progra*  is  intalTl- 
9lbla;*  or?  *a^  - Th*prodnBr  Is  aasy  ta  modify,*  v»ar* 
aacli*  Airttiar  anolytad  ta  aafln*  lass  abstraer,.  1 .r.  * 
aarrconcrata.  atsributas  caeabl*  of  baind  dlraetly 
waiiinrt  as  ta  >4iatnar  oi*  attrlbut*  Is  prasaire  in* 
softMi*  ta  too*  Jadi  ao  (air  a seal* of  3 to*  LOS).  A1> 
dmiqlT  a datallad  traakdOMir  of  aaeft  major  attrlbut* 
uas  il«dir  iir  tb*  rafarane*^  airly  a fa»  nacrlcs  '«ar«  d*> 
flaad  and  ns  parbtcular  aopileacio*  «*»  oanctonad. 

K Tatar  itudy^*^  parforwad  by  tb*  autbors  Includad 
tba  fbnailatloir  of  ratrlcs  and:  tbair  apol  fcadotr  lir  * 
codtrollad  axparlnant:  ta  a<a  caooutar  prodraan  (aO'- 
prox1a*taT]a4Qa  rdrtrair  staxaMnts  aaar)  tnoaoandancly 
praoarad  ta  tb*  sao*  spaclflcstloir.  Iir  tbis  study, 
only  * T1dtad,nuabar  of  aetributas  wnra  eonsidnrad. 
prlMrity  tbos*  eorrtspanalnd  ta  attnbutid  A$  and  A4 
of  Rnf,  3*  annttonad  pravlously. 

Dr  addItlORv  tbor*  was  * d*l  Ibarat*  diffbrane*  lir 
duality  Mbbasls  la  tb*  tna  arodranlnd  tfftrts:  on* 
H*s  don*  by  a *botsnat*  prosraanwr  nOo  mbs  ancauradod 
ta  oaxldlza  ead*aff1c1ancy,  and  on*  by  a earaful  pro* 
asaaar  «bo  tms  tneouradb^  ta  aaonaslt*  tlnal  fcir/. 
Tlwaila  rsaults  of  tb*  study  uara: 

* Tan  tlnas  as  my  arrars  ««r*  datactad  In  tb* 
'afflclant*  prod^w  (ooar  an  Idandcal  sarfas 
df  1003  tast  runs); 


• Til*  i*asiirns  df  prsdn*  duality  wnra  rt  dot  ft' 
CMtTy  hidbar  on  tb*  ’slnaT*”  prodmt  thus, 
tbay  Mara  good  Indicators  of  ralativ*  ooart'- 
tlonat  r«11ab1Tlty.  at  laast  In  drts  eantnxt. 

ConcurrantTy.  otbar  autbors  uara  racodnirind  tb* 
rldolfleanc*  of  cbaractarftlnd  and  dnallno  anlleltly 
witb  loftMr*  duality  attrlbutat.  'dilfla)  Idantlflad 
and  provldad  conds*  dnflnltlons  of  savan  inoortint  and 
ruajdnnbly  non-ovar1aoolnd  attrlbutas:  nalncalo* 
ahlTIty/aodlflablUty.  roouttnass.  clarity,  oarfor* 
aanea.  coat.  porttOlllty.  and  rnsaa  factors. 

Oamatby.  at  an^'  dafinad  a fiindar  of  ebaraetarls* 
ties  of  ooaratlnd  syttaa*  and  anniyznd  sona  af  tba 


tradaoffs  OatoMan  tba*.  walnbard^^  ptrfarnad  axpar+- 
mants  la  uOlcb  savaral  grouos  of  amqranmn  wara 
givan  tb*  san*  usldnnans  but  diffaranc  suceass  cri  ta- 
na (davalapannt  spaad.  nuoRiarof  statasants.  unount  of 
car*usad»  output  eUnty,  and  oradi^an  clarity).  For 
aactr  cbaractarlstle.  tba  hldnast  parfomanea  ms 
acblavad  by  tb*  groua  divaa  tbac  enaractaristlc  as  its 
suceass  cntaHon. 

Aa  Incraasind  nunOar  of  paoal*  Mara  raeodoitlnd 
tb* Inportinc*  of  softuar*  Quality,  and  ddallng  Im- 
plicitly ultb  softuar*  quality  attnbutas  oy  aoortsslnd 
tb*  astaOllshmanc  of  'good  prodruxnnd  oractlcas.*  ^ ^a 
boodoa  prodraawlnd  styl*  py  Karoignan  ana  Plaugarro) 

Is  tb»b«sc  txanqlaof  ttHs  Moru.  Also,  a sarlas  of 
raports  by  CIRAOI^’^m)  git  softuar*  naintainaoillty  pra- 
«1d*  SIM*  soure*  Tataria  I ana  son*  itans  suca  as  ui* 
falTowInd  cbwcklisc  of  oracrtcas  and  faaturas  tnnancind 
softuar*  aalntalnaolllty:  toncaptual  grpuslnd,  tao- 
doua  progi'Milnd.  idouianty.  'raanindfulnass,  uilfor- 
mlty,  compactnass.  natural nass,  cransfarablUty , cem- 
mants..  parantbasas,  ^nd  namts.  A raport  oy  lAarraa  on 
Sdftuar*  PortaOIHty^ll}  providas  aa  axcailant  susuiar/ 
of  altamatlvcapproacnas  ta  scrtaall  ir/--*.g.,  siwila- 
tloa,  nmlatloa,  intarpratattcn.  taocstrapoind,  nlgnar- 
oroar  landuad*  i‘aaeurfa— t*i  tr  pr'^ary  arenasis  aa  ai* 
portability  of  1anduad*orocasscrs. 

Nacandy,  a numOar  of  inltlatlvas  nav*  rscPdatzad 
ai«  inaercanc*  of  axplldtly  cansidarnd  duulity  fac- 
tors ia  softuar*  andlnaamd.  ‘tr  sxarola,  four  jro- 
iantatlons  la  tba  raeanr  AlAA/AC*/ lEEL'CCC  Softuara 
'lanadantnc  Cgoftranea  aoorass  tn*  suoject:  3ailoz»‘r 
PTOsantatlon'^l  idaotlfles  'seftuar*  luai  Ity  soaclfi- 
cadons  and  tradaefft*  as  a ngn-ononty  COO  fnlei- 
aclv*;  Kosslakoff^^l  Idantlflas  savaa  attclfciito* 

‘good*  saftMr*tpac1f1cac1ons;  '«01ta«ar‘-*l  icanrl- 
flas  tMlv*  asp II cl c duality  goals  for  nay^yCC  prodraw- 
afnd  landuadas;  andUgbe's  dfasanciclon«^'  on  sofr- 
Mtr*  duality  assume*  idantlflas  fiv*  important  mta- 
suraatof  softuar* quality. 

Kar  Tssuas  la  Sdftuara  Qualirr  gvaluatica.  S®* 
of  tb»<i«]or  issuns  in  saftMr*  dual : ry  avaiuatloa  ar* 
tb*  folloulnd; 

U ts  It  posrIOT*  ta  astibllsir  daf melons  of  tba 
cbaractarlstics  of  softuar*  quality  Moica  ar* 
maasuraol*  and  sufflcftncly  norr-ovariaopina  ta 
panrtt  softuar*  quality  tvaiuacicn?  This  Miii 
0*  dlseussad  la  Sactloa  II  oa  'Charactan sties 
of  Quality  Coda.* 

?.  Ha* Mil  eaa  ona  naasur*  ovaraTT  softuar* 
quality  or  Its  Individual  cbaractarlstics? 

TIHs  ullV  Oa  dlseussad  In  Sactlcn  HI  on 
"Itadurlnd  tb*  Qual  1 ty  of  Coda.  * 

3.  Mm  eaa  Infomatlan  oa  safttwra  quality  ebar- 
actarlstles  ba  usad  ta  lacrova  tb*  softuara 
11fa<yel*  praeassT  Tbit  u111  b«  dlseussad  la 
Saction  nr. 


Cod*  Is  tb*  raallzatloa  of  tba  toftMtra.rtaui'O- 
n*its  and  tba  datallad  softuara  dasign.  It  is  tna  ore- 
duetlo*  artlel*  that  dlraetly  eontrsis  tna  ooaratlens 
of  tb*  usar's  tysta*.  This  saction  of  tba  oaoar  aodras- 
SOS  tn*  proola*  of  cbaraetancind  tba  Quality  pf  tra 
csda  itsalf.  This  saction,  and  sna  followtnd  saetton 
on  naasurlnd  tba  duality  of  coda,  ar*  Msad  onrarrly 
on  a study  parfomad  on  wn.supjaet  by  TRW  for  tno  '-i- 
eional  Suraau  of  Standards,' (no  .tr  suosaeuont  .-:rt 
In  Oia  araa  at  TRU.  Including  a Saftuirs  AtHaOllily 
Study  porformad  for  Rona  Air  Davaioorant  Cancar.l*'' 


Initial  Stu^  Qblicfjvts  <na  Cancluslowt.  *>•  1"- 
Itlal  objtctivaj.V  '.rarscttrtt^cs  s’* 

Q»«lity"  study^-’^w«r«  to  ictntt^y  x st?  ai*  cnarictar- 
ijtics  saftwart  ouaiity  ina,  'or  sacn  tnarictans- 
. tic  to  oof  In#  on#  or  nor#  flotrlcs  suca  tnat: 

1.  Slvoit  an  aroltrary  snsqram,  th#  itotric  oro- 
»10##  a quantitativ#  roasuro  af  ta#  laqroo 
to  which  tn#  orcqram  tas  :n#  disaciatoo  cnar- 
actarfstlc.  anO 

Z.  OvoraH  softwar*  quality  can  bo  OofInoO  as 
sow  function  af  tne  values  of  to#  ■notrlcs- 

Alttnuqh  "softwart“  can  lav#  nany  coanwnants  suen 
as  functional  so#c1*l cations,  cast  3 Ians,  anO  aoara- 
clonal  manuals,  tnis  srjoy  concantratao  on  t#trics 
»n1ca  could  5#  appli#d  to  rartran  sourta  areqraas. 


finally,  w#  concluded  that  calculating  and  unoer* 
standinq  tne  value  of  a sinqla,  overall  necric  for 
software  quality  tay  oe  ■nor#  trouole  tnan  :c  *s  -ortn. 
T>ie  itajor  oroolam  is  that  tany  of  tne  indl visual  cnar- 
acterlstlcs  of  cuality  are  in  conflict:  aedeo  aff^ci* 
tncy  is  often  ourenasad  at  tn#  qr’ct  of  oortaoility, 
accuracy,  understancaoility,  ana  'naincainao<  i :ty;  iccec 
acruracy  often  conflicts  witn  oortaoilitv  /ia  oeoen- 
cance  an  word  site;  conciseness  can  conf’^ct  wlw  laqi- 
bllity.  Users  generally  fine  it  olff'cult  to  quantify 
tneir  preferences  in  sues  conflict  situations.  An- 
atner  proolaai  is  tnat  tn#  tecrlcs  are  generally  inccai- 
pleta  measures  of  tneir  assoetatad  cnaractarlscics.  To 
suaearlze  tnese  cons i aerations: 

I.  The  des Irani e qualities  of  a software  oroducr 
vary  with  the  needs  and  pr’oritles  of  the  aro- 
soectlve  user. 


The  Initial  conclusions  of  the  study  are  suewar- 
Ized  below.  "Irst,  in  software  product  savelopaent  and 
evaluation,  one  is  oenersily  far  itore  intarestsd  in 
where  and  how  rawer  tnan  now  often  tne  croouct  is  ie— 
f^cient.”T7ius,  the  tost  /aiuacie  aucoiratia  tools  for 
softwere  quality  analysis  wouia  generally  :e  those 
which  flagged  oeflctencies  or  ancsialies  in  the  program 
rather  than  just  proouctng  nuiroers.  This  nas,  of 
course,  been  true  in  tr.e  oast  *or  suen  itsms  as  com- 
piler diagnostics;  one  would  oe  jusclflaoly  irritated 
with  a mere  statement  wat  percent  of  your  state- 

eents  .nave  unbalanced  paren Weses.' 


2.  There  Is.  therefore,  no  singl|  aetrlc  which 
can  give  a universally  useful  ^ting  of  soft- 
ware quality. 


3.  At  best,  a prospective  user  coulo  receive  a . 
useful  rating  by  fumlsnlng  the  quality 
systam  with  a worouqn  sac  of  cnecxliscs  ana 
zriorntles. 


i.  even  so.  since  the  metrics  are  not  exhaustive, 
we  resulting  overall  racing  would  se  more  sug- 
gestive than  conclusive  or  prescriptive. 
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Second,  we  found  wet  for  virtually  all  the  slmole 
quantitative  formulas,  it  was  easy  to  fine  counttrex- 
iMiles  which  c-nallangeo  weir  creoiolHty  as  inolcators 
of  tof^re  quality.  Sane  cxasQles  are  given  below. 

1.  A metric  was  develcoed  to  calculate  the 
average  size  of  orogram  mooules  as  a meesure 
of  stmcBiredness.  However,  suaoase  one  nas 
a software  product  with  n lOO-statmnent  con- 
trol routines  and  a library  of  n S-staesment 
computational  routines,  wnicn  would  be  con- 
sidered well  structureo  for  any  reasonaole 
values  of  0 and  n.  Then,  if  n ■ 2 ano  m • ?8, 
the  average  macule  sice  is  5.3  statements, 
while  if  m ■ 10  and  n - 10,  the  average  .module 
size  is  S2.2  stataments. 

2.  A "robustness"  metric  was  develooed  for  the 
freetlon  of  statwents  wiw  potential  singu- 
larities (olvlee,  souare  root,  locariwm. 
etc.)  which  were  preceded  by  staemnents  wnicn 
tatted  and  compensated  for  slnguiarmtias. 
Mpwever.  often  we  ooentlon  is  in  a context 
which  makes  we  singularity  imoossible;  1 
slsvle  example  is  that  af  calculating  tne 
hypotamise  of  a Hgnt  triangle: 

2 • SQRT(X**2  ♦ Ym*2) 

3.  Sew  •lelf-descrlotlvenesi"  metrics  were  de- 
veloped 'or  the  nuNer  of  connant  cards,  we 
average  length  of  conaents,  ate.  However,  it 
was  fairly  aasv  to  recall  oregrama  wiw  fawer 
and  shorur  eonraitts  wnich  wert  much  aasier 
to  understand  tnan  tome  with  many  extensive 
but  poorly  written  caaments. 

Third,  we  concluded  Wet  We  software  'leld  is 
still  evolving  too  raoldly  to  estaaHsn  tetr'es  in 
sone  areas.  In  fact,  doing  to  would  ttno  to  rein- 
force current  prsctlco.  which  may.:^f  b#  goed.  'or 
tiample.  We  use  of  data  clusters'—’  wc  sucorntfie 
type-checking  would  invalidate  sc*e  -ei'ioility 
metrics  based  an  cheekihg  fer  -iveo-iuc#  axores- 
tlons,  paramatar  ••ange  violations,  ttc. 


5.  TharefPre.  the  best  use  for  metrics  at  this 
oolnt  Is  as  inolvidual  anomaly  indicators,  to 
be  used  as  guides  to  software  aevelooment. 
test  olanning,  acouisition,  and  maintenanca. 

Identification  and  CTassiflcation  of  Quality  Char- 
acteristics. raving  -sacheo  we  aoove  cone ius ion,  it 
was  oecidec  to  ceveioo  a nierarcnical  set  of  waraettr- 
i sties  and  a sec  of  anomaly-oetecclng  metrics,  jur 
plan  and  approach  were  as  follows. 

1.  Oef'ne  a sac  of  characteristics  wnich  are  ;.m- 
portant  for  software,  and  reasonaoly  exhaust- 
ive and  non-overlapoing. 

2.  Oeveloo  candidate  metrics  for  assessing  tne 
degree  to  wnic.*r  we  software  .nas  the  defined 
cherecteristic. 

3.  Investigate  we  characteristics  and  associated 
metrics  to  cecarmlne  their  correlation  wiw 
softwere  duality,  magnitude  of  potential  oene- 
f1ts  of  using,  ouantiflaoillty,  ano  aase  of 
aucomatlon. 

4.  evaluate  each  candidate  metric  with  resoect  to 
the  above  criteria,  and  with  respect  to  its 
Interactions  with  other  metrics:  overlaps, 
dependencies,  snortcomings,  etc. 

5.  Sased  on  wese  tvaluatlons.  refine  t.hc  set  of 
software  characteristics  Into  a set  whicn  is 
more  mutually  txclusiva  and  exhaustive  ano  suo- 
Mrtlve  of  software  Quality  evaluation. 

5.  Aefine  the  candidate  metrics  and  realign  wem 
In  tne  context  of  we  revised  sat  of  character- 
istics. 

ihe  following  Initial  set  of  software  characteris- 
tics were  develoeed  ino  oeflnec  as  a first  stto:  '*.} 

•,1)  Jnoerstancaoility,  2)  Ccmolateness.  3'  Concise- 
ness. (4)  ?ortao'iity,  '!1  Consistency,  \i]  i'linu'n- 
‘ioi’ity,  (7/  T«:i:i'ir/.  (31  ciaoiHr/,  '3'  ’el'ioi’- 
ity,  (10)  Structuredness,  (Hi  ifficiency.  definitions 
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of’  ttMS«-ci»r«ettrlst1es.  arm  givm  tit  tti»  Apptndlx. 


As. «.  sscand  $tS9»  w*  ttwr  itefintd  evSIdats  mss* 
sunMRts  of  Fartrut  cods  (t.s-..  nstrics.  vMiicii  Mould 
ssrw  as  ussfUl  tndlessorr  of*  tiir  cods' s Undsntand- 
•Slllcy.  Nslntalnaolllty,.  tte.  lit  Cotn^  so»  ms  found' 
thsc  any  sossurs  of  UnotntandaSIl  1 Cy  -mss  also  s mS" 
sursof  NsIntsInaSIllty— sines  any  cods  mslncananes 
rodttlrv  ttar  ttis  mslntslnor  unoontand  dis  coos.  On 
tfts  ottwr  iTsnd.  cnors  mots  nossuros  of  Mslntslnaol  1 1 ty- 
ttist  tad.  nothin?  to  do  Mittr  Undarstandaoillty.  For  tx- 
Mpltv  Ttstablllty  fisturor  sueir  as  suoaorc  of  lntsr> 
■talacs  outpuc  and  tcho-ctackln?  of  Inputs  ars  inoorw 
ems  to.  ths  rottsc  functloir  of  softusrs  mtlntSMncs*, 
buc  arsunrslatod  ts  Undontsnoaolllty. 

Tliur.  MS  bogsft  to  ffndr  ttist-  ttisetisnctv^sttes 
■ors  rolatad.  lir  s typs  of  tns  structurs.. 


In  Mirldr  ths  dlrsedos  of  ths  aiTq»  nprstants  c logo 
lesT  ttallcatlom  If  a progras  Is  talntxlnaols  It; 

\ asse  nocMsartly  bsUndontanoaoIs  and  Tsstafilo; 

i,  s.?..  sfrlgS  dogrosof  MtlntxinadlTtey  imifar  c 

MgIrdogrosaft-UndomandaOlllty  aos  rostaolitty. 


Ms  also  bogair  to  find  that  thors  mos  anothor  lavol 
of  i«rs  prlsltlvo  eoneaotx  b«lo»  ths  Itvoi  of  Undoro 
standablllcy  and  Tastablllty.  For  txanols.  if  a ?ro» 
gras  is  UndarstandaOlSi  it  Is  also  nteatsarily  Struc- 
curtdr  Conslsttnr,  and  Conelss  (thras  of  thr  original 
eharactartstics)  and  additionally  Logiblo  and  S*tf-uos> 
crfptlvs  (aw  additional  charaetart sties  not  impHad  by 
ths  thras  abovs) . us  Mors  thus  ganaratln?  toss  addi- 
tional charaetarl sties  and  finding  that  ais  antlrs  sat 
of  charaetarl  sties  could  bs  raprasantao  in  a tras  struc- 
turs»  In  Mhleir  each-  of  ths  mors  prisltivscnaractarls- 
tics  Mas  a naesuary  conol  tlos  for  sobs  of  ths  mors  gan- 
aral  charaetarl  sties.  (This  rasult  ancleloatad  stop.  5 
of  ths  plan-  outllnad  abovs. ) 

Ths  roaul  ting  Softwars  Qua.1  i ty  Charaetarlstics 
TVasis  shoMs  in  Fig.  I.  Its  Mghin^lavsi  ttruetiirs 
rafTacts  ths  actual  usas  to  Mhlen  tvaluaclon  of  soft- 
MOTS  duality  Mould  bs  put.  In  ganaral . «nan  pns  is  ac- 
dttlring  a toftJMra  pacicags.  ons  is  mainly  concsfriad 
Mltir  thras  guaatlons: 

- » HoismsIT  (aaalTy,  rallabiy,  affletantly}  eas 
L ussit  as-lsZ 

s ttasoasy  is  if  ts  maintain  (undarstand.  aod1^» 
andratast)? 


m Can  L stllT  uss  it  If  I changsny  anvlromincr 
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nws,  A*-«s  utility,  Milntalnatnity.  ind  ^oraBinty 
»rt  ntetssiry  'Sue  rot  suff1c1«nc  ) cenaltlons  I'sr 
lantral  Utility.  As-u  Utility  r«ou1ra*  i sregrtm  to 
.&•  #«li*Blt  4n«  Jdtfluattiy  tfficlant  ind  ■^uejnic.'igln- 
MTtd.  But  tOM  ■»ot  rwuir*  iit  as«r  to  ttit  trt  aro- 
gna,  undtrttind  its  inttmai  xonclngs,  rwalf:/  it,  or 
try  to  usd  it  tlstwidrt.  'i4lntain*oil1ty  reflu1r«s  wat 
th»ut«r  IM  aula  to  undarstana.  noat^y.  ana  test  tna 
prograa,  ana  Is  aldaa  oy  ;aod  liuman-anaTnearing,  jut 
doas  not  deoand  an  t.*ie  aroqr^'s  current  ?al  ’ sol  1 1 tt/ , 
Efficlanc/,  ar  »ortaol11ty  {exceot  to  tna  extant  t.na 
osar's  eoaputar  system  Is  undargolnq  avolutton). 

T>i*  lo*iar  Tav  1 stnictura  of  t-sa  snaraetari  sties 
tromorovidas  a sac  of  arlimtlva  cnaractarfsclcs  «nlcli 
art  also  strongly  ai"srent1ac»d  oitJi  resoact  to  eacn 
ottwr,  and  «nlcn  coffolna  into  sets  of  ■’acassar/  condi- 
tions for  tha  Intarwadlata-levai  cnaractan sties,  "or 
axaaala: 


• A prognm  wliIcA  doas  not  initialize  its  awn 
storag*  is  not  coaolataly  ial f •Cants Inao  and 
tOartftrt  1s  not  ccgnlataly  ’artsoia  avan 
tOougn  it  nay  da  comolacaly  Cavica-rnaacanoanc. 

» A arograet  using  fornats  sues  as  12.A6  Is  nor 
comlataly  3aylca-ineaoanoant.  ana  is  tSare- 
fort  nor  coaplataly  Portsoia,  avan  tnougn  it 
nay  da  coaplataly  iaif*Cantainad. 

a-  A program  wnice  Cavlca-lndaoandent  and  Saif- 
Contslnad  duC  Is  not  Aceuratt.  Scaoleta,  Ao- 
dusc.  Consistent,  Accountaola,  Cav’ca-Cffidant. 
Aecassibla.  Conminlcatlvt.  Salf-Oascriatlva, 
Stnieturad,  Condsa.  Ltgidla.  ana  Augmanuola 
stin  satlsflas  tha  dafinitlon  for  PortaoHity. 

Tht  prlsritiva  characteristics  thus  defined  provide 
a men  Sattar  foendaeion  far  defining  auantititiva- nar- 
rics  which  can  than  da  usad  to  naasura  the  tlatlva 
possession  of  both  tha  orisrltlva  and  tna  nlcntr  lava! 
eharactarl sties.  This  can  ta  done  in  tares  wnicn  aid 
both  In  avaluatlng  tha  utility  af  software  araeuets 
(at  tha  high  level)  and.  In  arwcriplng  dlraetions  o^ 
naadad  iworovamants  Uc  the  arimitiva  laval).  'Tie  def- 
inition and  evaluation  of  such  natrlcs  is  the- suojacr 
of  tha  next  Saetlort. 

nr.  "iaasurlnq  tha  Quality  of  Cade 


llatrics.  Tha  tarn  natric'*  is  defined  as  a naa- 
sura  of  the  extent  or  dagraa  to  wnicn  a product  (hers 
w«  art  concantrsting  on  coca)  oossassas  ano  axnibits 
a cartalH  (duality)  characteristic-  As  aaierioad  in 
the  previous  section,  wa  ^und  that  in  fact,  tha  aa- 
velopmant  and  ref''namant  af  aoth  aatrles  and  cnaractar- 
lattes  procaadad  concarrantly.  hany  aatr'cs  aooUcaOla 
ta  Fortran  coda  ware  fomilataft  and  analyttd,  leading 
in  several  iterations  doth  to  a refined  sat  of  imtrles 
and  to  tha  genera llzad  formulation  of  tna  niararenical 
sat  of  eharaetarlstici  oraiontad  ortviously.  u#  than 
had  a basis  for  avaluatlng  the  usefulness  of  those  on- 
tltlast  using  tha  following  criteria: 

1.  Carralatlon  with  Software  Qualit/.  for  each 
oetne  puroortaciy  taasur«ng  a gv/tn  'ar'iirf t1  va" 
charaetariatic,  did  it  in  fact  cor-alata  with  our 
notion  of  software  duality?  Hart  wa  :oaan  rougnly 

^Thai^an  subsets  of  aoolieatlons  'n  #n1ch  addi- 
tional characteristics  are  naeassary:  looltcitlons 
which  raduira  caaouttr  security,  far  axarole.  A more 
detallod  discussion  of  charsctarlstlcs  af  secure  sys- 
tmn  can  da  found  In  Aaf.  19. 


by  Ddsitiva  correlation  that  most  comuter  orograms 
with  high  scores  for  a given  metric  would  also  possess 
the  associated  prlmltlva  cnaractaristic.  Clearly,  a 
nora  precise  statistics!  Jafinition  could  sa  stated, 
put  tha  tvaluation  was  culta  subjective  at  this  pomt. 
ano  t.na  nert  precise  maasures  of  correlation  would  ntad 
tb  await  extensive  caca  cpi lection  ana  juegrents  on 
nany  diverse  esmouter  srograma.  The  following  scale 
was  usao  to  rata  eacn  racric: 

A - Vary  high  positive  correlation;  nearly  all 
arogrean  with  a nign  natric  scare  will  possess 
thm  associatad  cnaractaristic. 

AA  - High  oositiva  correlation;  a good  majority 

•'say  7!-9<3S)  of  all  prsgre*s  with  a nign  metric 
score  will  oossass  t.he  associated  c-iaractens- 
tic 

U - Usually  (say  50-755)  of  all  programs  with  a 
high  metric  score  will  oosseas  the  assadaced 
tntraeterlstlc 

■ - lame- programs  with  hign  .metric  scores  will  oos- 
sass tna  issoclacad  characteristic. 

2.  iQtentlal  benefit  of  metrics,  ioma  macr'es 
irovtaa  vary  ‘-tcrtanc  Msignts  sne  jec'slon 
:.-.outs  for  poth  :ia  savaiccar  ino  potential 
isers  of  1 software  proouct;  pt.iers  crova# 
•'nforBitlon  wn’c-n  is  •nteresemg,  Par*ios  in- 
clcatlva  of  cottntial  croolams.  cut  of  -o  greec 
'oas  If  tha  mttnc  does  nor  nave  a men  score 
*ven  thougn  Highly  correiatid  ws tn  its  associ- 
sted  duality  characteristic.  *he  jucament  is 
ta  Its  aotenclil  canefi?  is.  sr  ccursa.  aesen- 
jenc  on  the  'uses  'or  wnicn  the  evaiuaecr  is 
assessing  the  orooucr.  "he  following  scale  of 
pocontlal  danafits  wes  cafinod: 

5 - Extremely  imoortant  *or  .metric  to  have  a 
nign  score;  major  potential  trouclas  other- 
wise 

A - rjBortanc  for  metric  ta  have  a high  score 

3 - Pilrly  imoortant  for  metric  to  nave  a nigh 
scare 

2 - Some  Incremental  value  for  metnc  to  nave 
nigh  score 

I - Slight  incremental  value  for  metric  ta 
nave  high  score;  no  real  loss  otnerwlsa. 

3.  vatric  0uant1*i ability  arc  FeasiBlMr/  of  Auto- 
latao  evai-jat’on.  while  a matr-t  -ay  rata  at 
tna  toe  for  ootn  correlation  witn  oualitv  ano 
potential  oanafit,  it  may  be  vme-cansimiing  or 
•xoanaiva  to  catartrina  Its  nurerical  value. 

In  fact.  If  to  evaluate  a metric  'eoulres  an 
fxpart  to  read  a program  and  maxa  a juagmant, 
the  mmmrical  value  will  generally  orovloa 
much  lass  Ins'ghc  than  tna  uncerstandi no  that 
tho  axoort  will  pick  up  m tna  evaluatir.n  nro- 
:ass.  Furtharron,  metrics  reouiring  txoert 
insoactore  art  extravagantly  axcenslva  ta  'usa. 
"harefore,  one  would  orafar  for  large  programs 
tn  automatad  algorithm  which  tximlnas  tha  pro- 
gram and  produces  a metric  value  fine 
idly  also  a list  af  local  exceptions  to  posses- 
sion of  tna  relevant  cnaracttrlstics! . An  In- 
termediate cacacillty  wnich  is  often  tore  *ee- 
slble  is  an  automated  camel  lance  cnackar.  far 
wnich  the  user  -ust  orovida  a cntcxlist  of  ca- 
slred  duality  churactaristlcs. 


Iir  tli»«r&lu4tlofl»  <.  lutfgawic  VMS  iiad»  <s  ta 
Mirtcir  caMlMeloir  of  ottiraos  of  ouiflrfffcK-> 
tioir  would,  provldo  tn»  nose  easc>«fftetlv» 
rttiny  for  tlw  notrfe.  laln?  tn»  fanawlnr 
s«e  of  options:. 

AL<-  CIS  b«>doii*  eosCMffoetIvsIy  vis  <n  tuts* 
■stnd;  4l  port  tin 

CC  <-  cair  bMdon»cQtt*«ffoet1v«ry  vis  «it  wtOf 
■atsd  coapllanc*  chockor  If  givoir  s elwdc- 
Tlse  (Cod*  Auditor  1r  sudr  s tool,  oasoribotf 
lir  Soettoir  IV) 

Ut  - rtpairts  «r  untrslnwt  Irapoetor 

TT  - roqulrts  s trtlna*  Inspoctor 


EX  - roqulrts  an  txporc  Inspoctor 

EX  - roqulrts  proqrav  ta  b»  oxoeutod. 

Atttaasttnq'  so**  ovaluatlons,  sucn  as  countlnq 
avortqsaodul*  lonqtti  or  chockinq  for  cti* 
prtsonc*  of  ctrtain  kinds  of  sol  f-doscr1oc1v» 
■storlsl . cm  b*  don*  in  a fairly  tasy  and 
strolqhtftnMrd  fasnion.  Autosacln^  otnor 
tvolusdons,  sueir  as  scanninq  tno  oroqra* 
qloPoITy  forropootad  suPoxprosslons  anq  guar- 
antaslnq'  ttne  tft*-  coapononts  In  cn*  suboxoros- 
slons  ar*  not  nodlflod.  boovoon  roocdclons, 
ar» possIPI*  but  oor*^ difficult.  Otiiors,  suctr 
as  judqlnq-  tn*  daser1pclvon*ss  of  tn*  stlf> 
dairrlptlv* natarial  as  w*!!  as  its  croaoncot . 
arrwIrtUftTly  Inposslblc  tsaucorat*.  Thus. 


r«pr»r 

EVNJiATIQtt  OF  QUAUTY  NCntlCS 


frfwltlw 

OMrocttristlcj 

r OofInItloiT  of  Mttricr 

Corralstlon 

witir 

i^lltr 

Potancfal 

Sanaflt 

Ouantlfl- 

abllltv 

US*  or  .coBoiota- 
Oav*lop1iiy  i nass  of 
Autoftatad  j^taastad 
Evaludcion  rvaluation 

aaf*1c*> 

Indcpciidcficft 

Ot-l 

Arv^eaaputatlons  liidopondant  of  cem- 
putar  word  sit*  for  adrltvoMiic  of 
rooolrod  oroelslon  or  stono*  sent**? 

A 

£ 

AL  rer 
-Tt 

£ 

BTJ 

Hav»*ac»1n*Hlapoflaanc  ststwonts 
b*«r  flaqqad  and.  eowantol  (os.d;^» 
dns*  eoBPtttatlons  vdildr  dooond  upon 
eoivutar  hordHor*  cipadll Ity  for  ad- 
drossiny  half  words,  bytas.  saioctaa 
hltpaCtamr,  or  thosawnle*  «aaloy 
ntandad  sourc*  laiwuao*  foaturos)? 

AU 

(A 

5T?: 

Costal  n*dnoss 
SC-l 

Soar  01*  proqraa*  contain  a fadTIty 
for  Initlalltinq  car*  storaq*  prior 
trusaf 

A 

r 

Al;. 

6“ 

F 

SFT 

dots  tft*  proqrair  contain  a facility 
for  propar  pasldoalnq  of  Input/ 
outsut  davicas  prior  or  us*? 

A- 

5 

CC- 

e 

f 

Accuracy 

Ul~l 

At*  tft*  nuMdesT  natftodr  usad-  by 
th»praqraa-cons1stanr  witft  appli- 
cation raoulronants? 

A 

s 

TT 

— nrz — 

Ar*  tft*  aceuracits  of  oroqn*  con- 
stants aad  tabular  valuas  consls- 
tant  wItJr  aboHcatlon  rodulrsnnts? 

A 

s 

AL  “ tI 

£ 

? 

Ui^iotonoss 

CP-l 

Ara  alT  proqra*  Inouts  uaad  wltlrfn 
tti*  praqn*  or  tftair  pratanc* 
amlalnad  by  a esnimtl 

u 

X 

AL 

c 

C 

Ara  tftar*  no  'duaqr*  subproqnaK. 
rafironcadT 

1 

z 

AL 

t 

c 

AooiiStnosi 

R'L 

Ooas  tft*  proqn*  hav*  tft*  eaoabll- 
itr  ta  assign  dafault  valuas  os 
non-soadflad  oaramtarr? 

A 

s 

At*  n 

e 

f 

IS  input  data  cftackad  for  rang* 
trrors? 

AA 

§ 

ALr  U 

i 

p 

CMistney 

CS-l 

Ar*  all  tpaclflcations  of  s*ts  of 
gloftsl  varlabits  (l.a..  tftos*  ap- 
pasrlnf  In  two  or  aor*  suber*> 
gifts)  IdantleaT  (a.f..  Tabalad 

AA 

. s 

AL 

t 

c 

IS  tft*  typ*  i#.q.,  rtal,  fnttgar, 

ate.)  of  a varlabla  eonslstant 

for  all  usas? 

A 

"T 

"TC 

s, 

w 
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sem  lutaflatid  tools  could  imiat  ustf’jJ  out 
only  P«rt1»1  susoort  to  cutlir/  avtluation, 
iMvIng  t*t«  rwtaiRdtr  to  oe  suooiiad  Oy  i ru- 
mit  rt*a*r.  The  'ol'owin?  sal  as  >*ara  js«a 
I to  rat*  each  fietrlc  with  '■esoect  to  ees*  and 

coapitttntss  of  automatid  evaluation: 

Eas*  of  3*v*looino  Auteinatad  Svaluadon 

£ - Easy  to  develop  autonaced.  al<;or1thm  or 
coapi  1 ance-  checker 

N •Moderately  difficult  to  deveioo  aucsnated 
‘ aigprittm  or  comolfanc*  cneckcr 

0 • Difficult  to  deveioo  automated  algorithm 

or  coaol lance  checker 

CoMletenesa  of  Automated  Evaluacign 

C • Algorltha  or  checker  orovldes  total  evalu- 
ation of  metric 

P - Algorltha  or  checker  provides  partial 
evaluation  of  metric 

1 - Algorithm  or  checker  orovldes  inconclusive 

results 

Evaluation  of  vetries.  "he  adove  criteria  warm 
aooHed  to  tn*  tanaicat*  metrics  cove i coed  in  tre 
study.  Some  e-vamoiet  are  given  in  'able  I (alsplayeo 
on  the  previous  page).  Only  a small  fraction  of  tne 
151  candidate  metrics ^*°icoulo  Pe  induced  lere,  out 
.•  the  Ideas  are  anoly  illustrated.  In  Table  I are  snoMn. 
for  each  of  six  primitive  waracteri sties,  :.na  evalua- 
tion of  tMO  of  its  asspciatec  metrics.  An  exolanatlon- 
of  the  ratings  established  'or  the  first  metric  in  tne- 
. table  is  given  Oelow  for  clarity. 

The  metric  (OI-l)  was  found  to  Oe  highly  corre- 
1 lated  with  Device- Indaoenoence  frating  "A“),  ano  to  o* 
t extremely  'Ttortint  with  resoect  to  Device- Indeoendence 
(rating  ”5').  It  was  'ounc  that  a comolnatlon  of  auto- 
mated algorithm,  execution,  and  a trained  inspector 
r would  generally  be  most  cost-effective  for  determining 

I .  to  what  degree  a software  product  possessed  vi*  cnarac- 
**  terlstic  (rating  “AL  - EK  r Tr*}.  An  auconatad  aloo- 
rithm  coulo  checx  format  statanents  'or  aevi  ct-deo*n- 
f-  dance,  e.g. , or  F15.il  (more  precision  cftan  most 
! ! machines  possess),  and  slmHarly  -nag  extra-oreci s# 

••  constants  '’e.g,,  ?!  ■ 3.:dlS9265j£9).  These  cnecks 
wuld  Oe  easy  to  automate  (rating  out  would  only 
^ gravida  partial  results  (rating 

Evaluation  of  metrics  Versus  Prelect  Error  Experi- 
ence. The  pest  gooortunitv  to  evaluate  ae  metries  ana 
rating*  exoRplI'led  above  in  Table  I presented  Itself 
In  the  avallabillry  of  an  extensive  data  oasa  of  soft- 
^ I were  error  types  and  exeeri cnca  in  detecting  and  cor- 
recting them,  cbmolltd  by  TSW  for  she  Air  Force  :c:P-95 
study.  The  Initial  segment  of  this  deu  base  is  pre- 
sent^  In  Table  Z (disolayed  on  the  folloivlng  page). 

I ' It  Includes  the  classification  gf  Z24  softMr*  er- 
rors typed  into  13  major  cetagorl**: 

1.  Errors  In  preoaraticn  or  processing  of 
card  Inout  data 

2.  Tao*  handling  errors 

3.  Disk  handling  errors 

4.  Outbut  processing  errors 

$.  Error  message  processing  errors 

8.  Software  Interface  errors 

7.  Hardware  interface  errors 

I.  Data  base  interface  errors 

9.  User  int*r*ie*  errors 

10.  Cbwputetlon  errors 


11.  Indexing  and  subscrioting  errors 

12.  Iterative  orocedure*  errors 

13.  31t  manipulation  trrors 

in  t.h*  aarller  study,  all  the  arrors  of  aach  type 
were  analyzed  to  determine  Curing  wnlen  pnasc  of  t.h* 
software  oivelocment  process  they  were  tyoicaily  (Out 
not  always)  cormlttes  anc  wnere  way  were  tyoicaily 
(but  nor  always)  found  and  corrected.  The  pnases  used 
in  this  analysis  were: 

1.  Redulrements  3efin1t1on 

2.  Jesign 

3.  Cade  and  Cebug 
A.  Oevelooment  Test 

5.  Validation 

S.  Acceptance 
7.  Integration 
3.  Sell very 

In  Table  2.  for  each  error  type,  an  '0*  Is  placed 
In  the  Cbluaet  corresponding  to  wie  pnase  in  wnlen  that 
type  of  arror  r/pically  originated,  ana  an  "F"  is 
placed  in  the  colusm  csrrtsDOnaIng  to  ~h*  onasa  in 
wnich  that  type  of  error  i,as  v/pi tally  found  and  cor- 
rected. To  evaluate  t-ne  apol  icael  liry  of  aac.n  mccr-ic 
to  error  detection  ana  correction,  an  aoaltlonal  item 
was  estimated  for  esc.m  error  r/oe:  :.na  ones*  in  wnlen 
thar  type  of  error  «auid  most  likely  pe  cetectea  and 
corrected  using  »ac  metric. 

Example.  Line  I in  aie  table  'ndlcatcs  that  the 
cypicaTerw  of  ^i*  :yo*  in  «nicn  w*  program  expects 
a ptrameter  in  a dlf'erent  format  than  was  ghven  in  c.*.* 
Program  Aeculrement  Soecificaticn. ' originated  in  ci* 
3*sign  pnase  (at  least  for  the  software  projects  anal- 
yzed in  th*’  CCIP-aS  study).  That  type  of  error  wes  typ- 
ically corrected  curing  ue  aceeotsnce  testing  pnase. 
However,  if  metric  CS-13  (an  extension  of  mecrlc-SO-l 
covering  header  csnnercary)  nad  oeen  comoutad,  this  tyoe 
of  error  would  typically  have  pten  caugirc  and  corrected 
in  tn»  cesign  pnase. 

As  is  evident  from  this  exanple,  on*  result  of  tho 
analysis  was  to  identify  several  extensions  of  the  pre- 
vious metrics  wnica  would  oe  effective  in  error  detec- 
tion and  correction.  *or  exairol*.  the  CS-13  caoaollity 
cited  in  the  taol*  imolies  t.he  ncco  for  an  automated 
tool  which  would  scan  stancart  software  rodulc  header 
olocks.  It  would  cnecK  ne  consistency  of  cneir  tnfor- 
metion  witn  respect  to  assertions  in  tne  meeaer  blocxs 
aoout  the  nature  of  incuts  ana  outsuts,  including: 

* date  type  and  fOnnec 
e mmeor  of  Inputs 

a order  of  Inputs 

* units 

e acceptable  ranges 
e usocietad  startg*  locations 

* source  (device  or  logical  file  or  record) 

e access  (read-only,  restricted  access) 

The  CS-13  entries  In  the  table  indicate  that  if  coding 
of  Che  module  had  been  greccdeo  by  suen  a module  dc- 
tcrlDClon  with  the  assertions  about  its  inouts  and  out- 
puts, then  an  autsmatad  consistency  cneckcr  could  gen- 
erally have  caugnt  t.h*  error  before  coding  oegen. 


87 


TUf«£.  Ev4lu*t1oir  of  Error-Oettcting  CwOHItiti  (M«tHcs)  wr.  Error  Type 

(first.  12.  of  22t  error  types) 


SoftMrv  Phise 


Error  Type  ' — ^ 


Errors  in  Preparation  or  Procwsimr  of  Canf  rnout  Oac» 


I-  Proqraar  expects  oaraweter  in-  di  fferenc  fonrat  tftair 
is  given  in  Prograer  Pequireaent  Epeciflcarion^ 

Pfogranr  does  not  expect  or*  accept  t reouiret  paraneter. 

J.  Progrw  expects  peraneters-  in  a:  different  order  than 
thatMlrictr  is  specified. 


Progran  does  not  accept  dat&  through  the  entire  range 
•rfiich  is  specified. 

Prograei  expects  para«eter  in  units  different  from  that 
which-  Is  specified. 

Noalnar  or  default  value  utilized  by  orograe  in  the 
absence  of-  specific  input  date  is  differmt  froer  that 
whictr  is  specified. 

Prograer  accepts  date  outside  of  alTowable  range  T inits. 

Prograer  mITT  not  accept  alT  date  within  ariowebTe 
range  limits. 

Program  overfTows  core  tables  with  date  that  is  wfthin 
the  allowed  range. 

Prograer  overfTows  aTTotted  space  in  mass  storage  wi  th 
date  that  is  within  the  allowed  range. 

Prograer  executes  first  test  case  proqerfy  but  succeeding 
test  cases  fai  1 . 

Program  expects  parareter  in  a different  locaeioi*.  than 


Q 

CT-IJ 


Q ■ Error  origin  CS-N  ' Consisteney>check1ng  aid  N applied  at  this  phase  Miuld  generally  have  ( 
F • Error  found  teetad  error. 

CF’«ir  » Cdnplcteness-checiring-  aid  II  applied  at  this  phase  would  generally  have 


Tbble  2 


Of  the  1?  types  of  card  processlne  errors  shoMr  in 
TihTe  2.  the  CS'-Tl  consistency  checker  would  have 
irae^t.  S.  Overal  T . out  of  the  224  types  of  tirorr. 
this  caaablTIty  would  generally  have  caught  18.  The 
next  nett  effa^ve  capability  would  parfor*  checks-  on 
the  eonsistancy  of  the  actual  code  with  the  nodule  de- 
scription produced  during  tne  desier  pnase  for  caeablT- 
ity  CS-13  abovei.  (For  exaneTe.  for  each  output  asser- 
tion, it  would  cheek  if  tie  variahle  aoeeared  on  the  left 
of  an  oguals  sign  in  the  code,  and  perforw  e units  cheek 
on  the  eoimucation.)  This  capability  would  have  caught 
10  types  of  error,  but  not  tmtIT  the  inittel  code 
icaoning- phase. 

In  general,  as  is  saan  in  Table  3,  the  Consistency 
atries  were  the  most  effective  aids  to  oenertng  soft-, 
ware  errors.  Overall . they  would  hevo  eauot  24  of  tho- 
224  error  typos;  thoir  total  phasa  oain  ( vrm  sum  of  tho 
mneor  of  phasos  that  error  detection  was  advanetd  by 
tho  oKries)  anountad  to  89.  or  an  average  lam  of  2.9 
phosos  per  error  type.  Tho  next  oost  e^fsetivo  retries 
wore  those  for  Aobustnoss,  followod  by  Self-Contained- 
ness  and  Coanunlcativonoss. 


anoit  comEcnoN  EFfanvcNCS  cf  hetrics 


TBwr  t>OM| 

Mitrle/Prinitivel  Csrrietad  I Phue  fiain 
Characteristic  I (HoJ ! (Tatin 


Consistency  34 
Robustness  I 29 
Self-Containodnoss  19 
Comamleativemss  9 
Structuredness  I 2 
Sel  f-Oestn  oci  venesr  I 
Conciseness  1 
Accuracy  I 
Accessibility 1 


I » 

I i: 

I i 

! 4 

i 2 


88 


Th«  min  ■Tiessa<?e  3^  "abla  3 *ir' t la* 

allcacion  jr  iucs-^aiac  ina  siriiutcritaa 
^ooustness,  inc  ia ■ •— :r,T*i-*«c.‘’esj  " azz 

slam f’cinc  'rzryurir.z^  ‘n  sc*T<*ar»  ts’acfzn 

. ana  earracficn.  ’s  in  r.Tsarur.;  ::sc:usic.i.  aug 

‘ ' 1t  snoula  not  o«  too  suroris'tng,  sines  Consistsney. 
^uatnass,  and  £«lf>Containeaness  are  tnrss  jr  ;ne 
priaritlve  cnaracteri sties  associatea  -itn  ^aliaoiUty. 

Anothtr  ussfu]  eensistancy  ensek  «as  '.a  esnoare  a 
attrte's  arror-cer^ction  eotentlal  -i?n  tne  astisate 
0/  a nstric's  eotsntial  eenefit  In  "aole  1.  iacisi'ac- 
torily,  virtually  all  if  tre  si^nli'icant  ar’~sr-aetsc- 
tin^  and  carracting  netnes  nad  -aaxinun  aotantial  aane- 
flt  rarfngs  of  3,  and  non*  «nicn  contridutsd  to  arrar 
coiTtetion  had  ratings  less  than  3. 


0 S*tt1nq  axolicit  softv»*r«  auallty  objectives 
and  priorities; 

0 Usinq  saftivere  quality  checklists; 

0 Sstablisninq  an  axoHcit  quality  assurance 
activity; 

0 Using  quallry-ennancing  tools  and  teennioues. 

f • 

Ex^ieit  Softivara  ?ual*tv  3iii actives  and  ^^er- 
ties.  'The  axoer’nienis  "aported  in  J.s’'s.  a jna  7 
sn<M*d  that  tne  tegrae  of  quality  a eerson  outs  into  a 
progran  correlates  strongly  vltn  the  software  quality 
objectives  and  on  or*  ties  ne  nes  seen  jiven.  Tnus, 
if  a user  wants  ocrtioility  ano  remtainability  nor* 
than  sod*  efficrency,  ft  is  'TOortanc  to  tell  t.ne  se- 
veloper  yiis,  ore'erioly  in  a way  wnich  allots  the 
user  to  Petemine  to  wnac  extent  these  qualities  arc 
present  in  tn*  final  oroduct. 


Used  IS  an  acctotancc  test,  software  suaifty 
benchmarking  jrovioes  an  exolicit  set  of  qua’'ty  objec- 
tives for  all  oartic'sants  througnout  the  various 


onases  of  the  software  ieveloonent  cycle.  It  has  oeen 
-sac  ontnarily  to  cate  in  soecifying  nareware-software 
'••I'acility  and  avai’amlify  oojectives  .wiqn  carticu- 
'ar  success  m tr#  jell  ^aos'  ilactronic  Switching  S/s- 
tam,  *or  examole).  out  nas  also  oeen  jsea  successfully 
*op  other  cuality  oojectives. 

Software  quality  ienc.imarking  can  and  snould  5e 
used  also  to  evaluate  alternative  software  srocucts  'or 
crocurement.  in  coing  so,  tn*  level  of  effort  exoaroeo 
in  ouality  Oenc.nmarxing  snoulo  o*  orooort'' cnal  to  the 
amount  ;f  use  the  orccuct  is  excecteo  to  nave,  gather 
than  tc  'ts  trice,  ^'ore  than  we  once,  we  "ave  lost 
over  S 10. SCO  in  soft.vare  oeveioomenc  and  ocarationai 
costs  oeciuse  of  •ncomolete  cuality  oencrrarxmg  in 
procurement  of  S1,CCQ-S2, jCQ  software  oroducts. 

Software  OuaVzv  CheckPsts.  The  quality  netrics 
suananzed  aoove,  aro  cresenteo  in  teeail  in  Sef.  15. 
can  oe  usee  as  the  oasis  'or  a set  of  software  cuality 
Checklists.  These  can  Oe  used  to  supoort  reviews, 
walkthrcugns,  insBections,  and  other  constructive  fn- 
teoencent  assessnents  of  a software  cevei cement  oroo- 
jct.  -gain,  to  cate,  "hese  nave  oeen  used  :r*rar'Ty 
to  suocort  software  'eliaoil'ty  oojectives.  out  tan  oe 
•used  effectively  'or  other  software  cuality  oojectives, 
-or  exaaol*.  "aole  - tives  a portion  of  a ohecxUst  'or 
judging  the  set f-'escrictiveness  of  a ocrcuter  orocram. 
an  imoortant  cnaracter'stic  n evaluating  tie  long-tem 
costs  of  understanoing,  testing,  and  saintaining  t.he 
program. 

”aOl*  4 

PaRTTAL  CHECKLIST  "R  JUOGING  THE  ja.-- 
OESCRIPTIVE.MESS  OF  A SOFTWARE  PRODUCT 

A software  oroouct  oossesses  seir-oesenotiveness 
to  th*’ extent  that  it  contains  enougn  information  'or  a 
reader  to  determine  or  verify  ‘t$  oojectives,  assuas- 
tlons,  constraints,  inouts.  outputs,  cotneonents , anc 
revision  status.  Checklist: 

a.  Does  each  program  nooule  contain  a neader  block  of 
commentary  wnich  describes  (1)  program  name,  {2)  af- 
fective date,  IS)  accuracy  recuirement,  (il  ouroose, 
(5)  limitations  ano  restrictions,  (5)  ooolf' cation 
history.  (7)  inouts  and  putsuts,  (3)  metncd,  !3!  as- 
simutions,  (ID)  error  ’■ecovery  procedures  fsr  all 
foreseeable  er^r  exits  that  exist? 

0.  Are  decision  points  and  suoseouent  branching  altern- 
atives adequately  qescr'bed? 

c.  Are  the  'unctions  of  the  -nooules  as  well  as  inouts/ 
outouts  adeouately  defined  to  allow  moouie  testing? 

q.  Are  comments  provided  to  suooort  selection  of  soe- 
clfic  inout  va’ues  to  permit  aer'ornenc*  of  soe- 
pialized  sroaram  testing? 

e.  Is  infomacicn  orovioed  to  suooort  assessment  of 
the  Imoact  of  a change  in  other  eortlons  of  the 
program? 

f.  Is  information  provided  to  suooort  identification 
of  orogrim  coca  wnich  wiust  be  modified  to  effect  a 
nequireo  change? 

g.  Where  there  is  ncdul#  deoendenc*.  1$  it  cleerly 
specified  ay  caerentary,  arogram  documentation,  ar 
•nnerent  arogram  structure? 

1.  ire  variable  natres  aescriotlv*  of  the  anysical  or 
'unctlonal  prooerty  reoresented? 

i.  Co  uniouely  recognizaole  functions  contain  aoeouatc 
descrlotiv*  in'ot—jtion  (e.g.,  conments)  so  that 
the  ouroose  of  eacn  is  clear? 

j.  Are  ia*<iuatt  descriatiors  orov'aed  ta  allow  earr*- 
lation  of  variiol*  *im*s  with  tne  anysical  arooerty 
or  entity  wnich  they  'merssent? 


Of  course,  this  was  Just  a aarclal  evaluaticrr,  but 
since  testing  occupies  such  a greet  procortion  bf  a 
total  software  effort,  tne  above  aveiuafon  has  been  a 
most  useful  one  for  leloing  us  to  decide  wnich  retries 
should  have  hign  priorities  for  ceveiooment  and  use. 

'We  hpvc  subsequently  ceveiooed  some  of  these  retric- 
checkers  and  used  them  with  some  success,  and,  in  par- 
ticular, the  CS-13  cetric  was  usad  as  a basis  for  the 
recently  develooec  Cesign  Assertion  Consistency  Checker 
described  in  Ref.  2D. 

lY.  Using  Quality  D.haracter'stics  to  rmorove  t.he 

^ftware  '»-C/c'.a 

The  software  life-cycle  orocsss  pegins  with  a sys- 
tem and  software  rsduirements  determina^on  prase,  'al- 
lowed by  successive  pneses  'or  system  cesign,  detailed 
design,  coding  ana  testing,  and  culminating  in  an  oo- 
erttions  and  maintenance  pnase.  "here  are  'our  major 
amns  w*  have-  'aund  far  using  t.he  auality  cnaracteris- 
tlcs  discussed  aeove  to  imorov*  tne  life-cycle  arocess. 
These  are: 


Probably  the  aest  way  to  accomolish  this  to  date 
is  through  the  practice  of  software  pualitv  bench- 
mi  ritinq.  Senenmarking  is  general '/  ;ust  usco  to  de- 
termine Devlca-Ef'iciency  on  i tyoical  oceritional  aro- 
f11*  of  user  Joos,  but  it  tin  be  used  similarly  ::  an 
acceptance  test  or  software  package  selection  criter*on 
for  other  oual tries  also.  "hus.  'ar  melntalnaaility, 
on*  constructs  a reoresentativ*  coeraticnal  qrafil*  bf 
likely  modifications  to  tn*  software,  and  -easures  now 
efficiently  and  ef'ectively  tnes*  -oolfications  are 
sMd*  by  the  actual  software  maintenance  bcrsonnel . 
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^|1ty  Assurance- Activity.  We  »re  finding  It  Ifr- 
ermsln^ly  advancageaus.  froiToottr  ortducc  au*lity  and. 
cost-«/^act)vene4s  standpaines,  ca  naveae  exoHcft 
quality  assurance  activity  orr  our- software  orojects. 
TheMnaqar  of  tftir  activity  generally  reports  co  ttie 
project  nanager*  and.  Is  ouroosely  held'  front  produc1n9. 
any  of  the  deliverable  product  In-  ordei*  to  provlae  air 
Independent  view,  of  the  project.  Taske  Included  In 
this  quality  activity  are  tailored  tsr  the  project  and 
depend  upoir  the  size  and-  scope  of  the  project.  This 
approacir  har  proven*  effective  In  ensuring’  that  the 
project  Is  responsive  ta  the  quality  reoul rarencs. 
of  the  customer  and.  the  particular  system  application. 
The  responsibilities  of  the  quality  assurance  activity 
geiwrally  Includer 

e PTamrlnr  - Preparatfoir  of  e software  quality 
assurance  plan  whicir  Interprets  quality  pre- 
grae requirements  and  assigns  tasks,  schedules 
and.  organizational  responsibilities. 

e Pol1qr»  Practice  amt  ProcedureOevelopmant  - 
Preparation  of  standards  manuals  for  all  phasas 
of  software  production.  Including  requirements, 
design.,  coding,  and  test.,  tailored  ta  specific 
project:  roqulrenents.  A key  point-  here  is  at- 
taiMon  to- quality  provisions  early  In  the 
software  1 1 fe  cycles. 

as  Software  (^lallty  Assurance  A1  dr  OeveTooment:  - 
Adaptatlair  and  development,  of  nanual  and  auto> 
aatW  procedures  for  verifying  compl  lance  cr 
software  flinctlonal  and  oe^armance  require- 
—Its  and  project  qual  Ity  standards. 


Quallty-ghhancino  Tools  and  Technioues:  -Peoul re- 
gents ana  cesian.  Junng  requirements  and  design 
Phases,  some  assurance  Chat  desired  quality  character- 
istics are  present  can  be  ootalned  ay  using  guidelines 
and  detalTed  checklists.  Here,  of  course,  the  primary 
objective  is  not  to  ootain  a numerical  measure  of  the 
extent  to  which  a quality  characteristic  Is  present, 
but  ta  Identify  problems  of  varying  levels  of  critical- 
ity with- whicir  we  need  to  deal.  A great  deal  of  soft- 
ware quality  leverage  Is  gained  by  using  machlne^anaiy— 
table  software  specifications  and-  automated  aids  to 
analyze  them- i^br  Consisiaqcy,  Camoietaness,  etc.,  suen 
as  [SOOS(ZI)  and  SREP^^^*^,  orovlce  for  software  re- 
quirements. These  systems  srovicea  significantly  In- 
creased assurance  of  specification  quality  over  manual 
methods,  giving  considerable  laverage  for  1 1mrclng  the 
cost  of  software  errors  during  eoth  testing  and  main- 
tenance. Thir  Is  because  a majority  of  errors  arise 
owing  ta  fau  I ^.expression-  of  requirements  ana  Incom- 
plete desIgnl^^J , and  are  much  less  expensive  ta  cor- 
reed  during  the  early  phases  of  production  than-  durlng- 
subsequant  phasas. 

One  of  the  biggest  sources  of  software  problems, 
stems  from  amoloulty  In  the  software  reoul rements  spe- 
cif IcatlonsT'T’TtmSer  of  alfferenc  grcaos— designers,  . 
testers,  trainers,  users— must  Interpret  and  operate 
with  the  reguirenents  inceoendently.  If  their  Inter- 
pretatlans  of  the  reoul  rements  are  olfferent,  many  de- 
velopment and  operational  proo terns  will  result. 

'jn»af  the  best  counters  to  tiris  prcbleRr  Is  e re— 
v1e»  tff  make  sure  that  the  reoul re-^nts  a^  Testable. 
Par  example,  consider  the  pairs  of  saecificatlans  belmK. 


» Audits  - Review  of  project  procedures  and.-  docu- 
OMtatlon  for  eomoTlance with  sdftMrer develop— 
malt  plan  standards,  with  foIToieua  and  docu-  - 
maitatloir  of  corrective  actions, 

e»  Tint  SiirvelTTance  - Reporting:  of  software  prob- 
Taa,  ansTysIs  of  error  causes  and  assurance  of 
carrectlve  actloir. 

• atords  Retention  - Setaitfoe  of  design  and 
ssftwere  problee  reoortr,  test  cases,  test 
dita*.  Togs  veriiying  quality  assurance  revlewr 
Md  other  actions. 

• Physic*!  Hedi*  Control  - rnsoectloir  of  disks, 
tapes,  cards,  and  other  prograi  retaining  medf* 
fa*  verification  at  alT  times  of  physical 
transmittal  or  retention,  amt  assurance  that 
CDOtents  are  not  destroyed  or  al  tend  by  an- 
sironaart  or  elshandllng. 

Vt  dete.  most  of  these  nsponsibl Titles  involve 
eonsldentlons  qf  reliability  and  software  proouct  cosi- 
pTIance  to  standards  and  the  software  raoulrenants 
specification.  However,  the  quality  assurance  activity 
can  also  bee  usaful  focal  point  for  assuring  that  the 
product  possessas  the  other  characteristics  of  soft- 
mnvqaellty,  such  as  Maintainability  and  Portability, 

(l>ieT1ty«€nhanc1nq  TooTs  and  Techniques.  Precti- 
ealTy  every  loftwan  tool  and  tecimioue— cross-refer- 
enee  generators,  flow  charten.  configuration  managt- 
mmt  procedures,  softwan  wnltors— sucoorts  some  kind 
of  quell tyanhanemnent  with  respect  to-  at  least  one 
quellty  characteristic.  Ken  we  concentrate  on  several 
tools  and  techniques  having  particularly  nigh  leverage 
for  softwere  quality  enhancament,  first  those  which 
apply  to  softwre  requirements  and  design  specifica- 
tions and  then  those  which  apply  to  cede. 


MowwTestable  Testable 


I.  Accuracy  snail  be 
sufflcienc  ta  supporc 
mlstioir  planning 

SystasrsnalT  provide  ZL 
real-time  response  to 
status  queries 


£ renrinate  the  Simula-  7. 
tioit  at  an  appropriate 
Shift  break 


t.  Position  error  shall  ber 
iSO*  In  the  horizontal 
X 2QT  1e  thar  vertical 

System- sheir  respond  tts: 
Type  Af  queries  fir  « £ 
sec  “ 

Type  C queries  1e  a.  Id 
sue 

lypsC  queries  1e  <^Z 
Ulft 

TcmlnaCe  the  slnuTation 
after  S hours  of  simulated 

time 


It  Is  clear  that  the  specifications  an  the  right  are  not 
only  ’aore  Testable  but  also  less  urelguous  and  aetter 
suited  as  a baseline  for  dtsigning,  costing,  documen- 
ting. operating,  and  maintaining  the  systme. 

There  Is  one  technique  for  explicitly  analyzing 
such  quality  considerations  during  the  reoulremcnts 
phase  wfllch  has  been  reasonably  successful  on  smail-to- 
"edlimi  projects.  This  it  the  Reou  l rements  - Properties 
Matrix:  * matrix  whose  column  consist  of  cne  Individual 
f'jnctlonal  reoulrementr  and  wnose  rows  consist  of  the 
major  qualities  (or  properties)  desired  In  the  software 
product  (or  vice  versa).  The  elements  of  the  .matrix 
cpnsisc  of  additional  specifications  which  ante  when 
qne  cpcsiacrs  tnu  quality  Implications  of  each  reoul  re- 
tent.  Porexaxmle.  consider  the  third' pair  of  rpduire- 
ments  tbove  wnen  treatad  in  a Reoul renena-Preoertlta 
.‘fatrix  It  In  Pig.  2.  It  Is  clear  that  the  rtmltiwf 
specifications  will  lead  to  a nigner  ouelity  teftwere 
product.  Some  additional  affort  would  be  nucettarv  *• 
achieve  the  tnnaneed  product,  but  if  cue  prpfram  m it 
have  a good  deal  of  use  and  maintenance,  tme  tf^eet 
will  pay  off  In  reduced  Ilfe-cyele  eusu. 
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Z - Portloir  of  a Rtqu1r«nnts-Prop«rt1ts  Matrix 


■■-"*.i*,^RA9u1r«BAnt 

Testability 


Modifiability 


Robustrass 


Terwlnata  th*  simulation  ' 

at  an-  aopropriata  snift  ottakl 

Tanrinaor  the  simulation  I 

after  8.  hours  of  simulated,  i 

I 

AlloM-user  to  specify 
temlnatloir  time'  as  an  In- 
put parweter.  with  a de- 
fault value' of  8 hours 

Provld*  an-  alternate 
termination  condition  in 
case*  the-  time  criterion 
caimot  b*  reached. 


QualltyEnhandno  Tools  and:  Technioues;  Code. 

As. shown  In  the  detailed  characteristics  tret  of  .-fg. 

1.  code  Structuredness  Is  one  of  the  necessary  primi- 
tive characteristics  for  Testability.  Understanoabll- 
1^1  or  Modifiability;  all  of  the  latter  are  necessary 
for  Maintainability.  A set  of  allowable  Fortran  con- 
structs for  the  basic  control  structures.ScQUEnCE. 
IFTMOIELSE.  CASE.  OOWIILE.  and  OOUNTTL^<-^I  were  devel- 
oped as  a standard  for  Structuredness  on  a large  real- 
time  software  project.  An  automated  Fortran*  source 
code  scanning  program  called  STRUCT  was  developed  and 
Is  regularly  used  ta  determine  for  each  routine  whether 
It  Is  a properly  nested  combination  of  the  allowable 
constructs;,  and  whee  violations  are  recognized  the  code 
causing  the  violatloir  Is  Identified  and  a.  diagnostic 
Issued. 

The  discipline  Invoked  by  this  quality  reoulre- 
ment  on  the  particular  project  met  with  a certain 
amount  of  resistance  and  disgrunti ament  by  program- 
mers. Functional  team  leaders  were  somewnat  dismayed 
at  first  since  routines  previously  coded  before  the 
standard  had  been  required  needed  to  be  redone  to  a 
great  extant,  which  consequently  strained  labor  cost 
budgets  and  made  the  original  schedules  difficult  to 
meet.  Subseouantly.  however.  In  a survey  of  program- 
mers and  their  supervisors,  most  were  of  the  opinion 
that  nelntenance  costs  would  be  reduced.  In  addition 
to  expressing  positive  opinions  on  "quality  of  code* 
Including  consistency  and  unoerstanoablllty.  The 
opinion  was  also  expressed  that  had  the  standard  been 
Invoked  In  the-  first  place,  most  of  the  development 
problems  would  have  been  avoided. 

Subsequent  evaluations  of  software  errors  ob- 
served In  testing  on  the  referred-to  oroject  have 
sboMP  an  extremely  low  rate,  believed  to  be  oartly  at- 
tributable to  the  application  of  the  Structuredness 
standard,  although  the  requirement  to  exercise  all 
branches  of  a routine  prior  to  turnover  for  Integration 
testing  was  found  to  be  much  more  Influential  In  reduc- 
ing the  number  of  errors. 

The  structuring  standard  Is  simply  one  of  over  30 
other  coding  standards  formulated  for  this  software 
project,  and  automatically  checked  by  a Fortran  source 
code  analysis  tool  called  CODE  AUDITOR.  This  tool  de- 
termines whether  these  standards  are  violated,  and 
shows  the  source  code  location  of  the  violation  In  a 
diagnostic  printout.  Many  of  the  or'*mitive  character- 
istics of  Fig.  1 are  measurable  by  CCOE-AUDITOR.  For 
example,  Self-Oescriptlv'-ness  Is  measured  in  part  by 


cheeking  for  the  presence  of  stendard  module  header 
esmsantary  cards,  and  for  coeemneary  cards  to  explain 
transfers  of  control.  Consistency  Is  measured  In  part 
by  checks  for  mixed-mode  expressions,  and  compliance 
with  standards  for  parameter  passing. 

In  the  future,  some  of  these  will  be  done  more  ef- 
ficiently through-  standard  language  features  for  struc- 
tured prograsMlng,  data  typing,  etc.  Yet  there  will 
remsln  a need  for  automatic  post-scanning  of  code  to 
assure  compliance  to  local  standards  (e.g.,  naming  con- 
ventions) and  to  check  for  partial  Indicators  of  poten- 
tial quality  probimn. 

Of  course,  as  Is  evident  ffom  the  ratings  fn  Table 
I.  there  are  some  evaluations  of  code  (and  design) 
quality  which  require  trained  humans  to  perform.  Ref- 
erence 25  reports  some  experiences  on  large  software 
projects  with-  emphasis  on  the  relative  merit  of  a var- 
iety of  techniques  involving  both  human  insoectors  and 
automated  tools  for  controlling  and  measuring  software 
quality.  One  particularly  valuable  technique  Ib.tblt 
regard-  Is  that  of  the  design  or  code  insaect1on^‘°l 
and  its  "cousin!  tK.*in1qua.  the  s^jctured  -wafTthrough. 
In  our  analys1s‘^' ' of  software  errors  *ouno  <n  we 
validation  phase  of  one  large  project,  «e  oetermlned 
that  58  percent  of  them  could  nave  been  eliminated 
early  by  approprjate  design  inspection  techniques. 
Fagan's  results^^^SJon  one  fairly  well -control  led 
project  Indicate  cnat  the  extra  effort  Involved  In 
oerforming  inspections  paid  off  In  a 23  percent  re- 
ouenon  In  operational  errors. 

V.  Conclusions 

ExollcIC  attention  to  characteristics  of  software^ 
quality  can-  lead  to  significanc  savings  In  software 
life-cycle  costs  (Section  I). 

The  current  softwere  stete-of-the-art  Imposes  spe- 
cific limitations-  on  our  ability  to  aucamatlcally  and 
quantitatively  evaluate  the  quality  of  software 
(Section  II). 

A definitive  hierarchy  of  well-defined,,  well -dif- 
ferentiated cheracterl sties  of  software  quality  nas 
been  oeveloped.  Its  higher-level  structure  reflects 
the  actual  uses  to  wnich  software  quality  avaluation 
woulo  ba  put;  Its  lower-level  characteristics  are 
closely  correlated  with  actual  software  metric  evalua- 
tions which  can  be  performed  (Section  IXJ. 

A Urge  mmrtwr  of  software  quality-evaluation 
metrics  have  been  defined,  classified,  and  evaluated 
with  respect  to  their  potential  benefits,  ouantlflabll- 
ity,  and  ease  of  automation  (Section  III). 

Fartleular  software  life-cycle  activities  have 
bean  Identified  which  have  significant  leverage  on 
softwerr  quality  (Section  IV).  These  Include: 

» Setting  explicit  software  quality  objectives 
and  priorities; 

e Performing  software  quality  benchmarking; 

e Using  softwarm  quality  checklists; 

•>  Estibllshlng  an  axpllcit  quality  assurance 
activity; 

e Using  macirine-analyzabla  software  specifica- 
tions; 

e Ensuring  testable  software  rtqulremtnes; 


» Us1n9  • Rtquirerants-Proptrtits  HatHx; 

» EsUbllshIng  sundarts,  particularly  for  stnic- 
turtd  coda; 

a Using  an  autonated  Coda  Auditor  for  standards 
coapllanca  chocking; 

ft  Parforming  dasign  and  coda  Inspactlons. 

Host  Importantly,  we  balleva  that  the  study  re- 
portad  In  this  paoer  provloas  for  the  first  time  a 
clear,  well-defined  framewonc  for  assessing  the  often 
slippery  Issues  associated  with  software  quality,  via 
the- consistent  ana  mutually  supportive  sets  of  defini- 
tions, distinctions,  guidelines,  and  exoenencos  cited. 
This  framework  Is  certainly  not  complete,  out  It  has 
been  brought  to  a point  sufficient  to  support  the  eval- 
uation of  the  relative  cost-effectiveness  of  prospec- 
tive code-analysis  tools  presented  In  this  paper,  and 
to  serve  as  a viable  basis  for  future  refinements  and 
extensions. 

Befe rentes 

1.  CIshoff,  0.  L.,  "An  Analysis  of  Some  Commercial 

PUI  Programs."  IEEE  Trans.  Software  engineering. 
June  1976,  ?p.  113-120. 

2.  Improvements  Needed  In  Hanaolno  Automated  Oeclslon- 
maklnq  by  Coirouters  Througnout  the  fecerai  Covern- 


went.  U.S.  General  Accounting  Orflce,  April  2o, 

I37f. 

3.  Rubey,  R.  J.,  and  R.  0.  Hartwick,  "Ouantitative 

Measurement  of  Program  Quality,"  Proceedings , 

ACM  Rational  Conference.  1968.  pp.  671-677. 

4.  Browm,  J.  R.,  and  H.  Ulpow,  The  Quantitative  ^*ea- 

surement  of  Software  Safety  ana  Rellaoilitv.  re- 
vise from  TRW  Rmrt  ho.  S0P-17V6,  August  1973 . 
TRH  Software  Sergei  (In  press  August  1976). 

5.  Uulf,  W.  A.,  "Programelng  Methodology,"  Proceed- 

ings of  a SymposluB  on  the  High  Cost  of  Soft- 
ware. J.  Goldberg  (ed.),  SUnford  Research  In- 
stitute, September  1973. 

6.  Abernathy.  0.  H.,  et  al,  "Survey  of  Design  ^als 

for  Operating  Systems,  "GaorglainstltuteorTecn- 
nology  Report  gVi5-72-04. 

r.  deinberg,  G.  M.,  "The  Psychology  of  Improved  Pro- 
graewer  Perfoneance."  Oa tarnation.  Hovember  1972, 
pp.  82-85. 

8.  Kemlghan.  8.  M.,  and  P.  J.  Plauger,  The  T aments 

of  Prooraulnq  Style.  McGraw-H1 1 1 , 1974. 

9.  A Study  of  Fundamental  factors  Underlying  Sof^p 
"FSintwance  problems,  cimo.  Inc.,  iiecember  1971. 

10.  Research  Toward  Wavs  of  Improving  Software  Main- 

tenance. CIRAO.  Inc..  January  1973. 

11.  darren,  J..  Software  Portability.  Stanford  Univer- 

sity Digital  Systems  Laooratory,  Tecnnical  Note 
Re.  48.  Septamoer  1974. 

12.  OeRoze.  8.  C.,  "000  Defense  Systme  Software  Menaoa- 

nant  Program,"  Abrideed  Proc^lnos  from  the  Soft- 
ware Hanaoeayt  Confyence.  1976  (ootainaele 
through  Los  Angeies  ^tlon  AIAA). 

13.  Kosslakoff,  A.,  and  T.  P.  Sleight,  "Software  Re- 

quirements Analysis  and  Validation,"  Ibid. 


14.  Whitaker,  W.  A,.  “COO  Coefton  High  Order  Language 

(HOC)  Program,"  Ibid. 

15.  Light,  W.,  “Software  Rellablllty/Quality  Assurance 

Practices,"  Ibid. 

16.  Boehm,  8.  d.,  J.  R.  Brown,  H,  kaspar.  M.  Lloow. 

6.  J.  MacLeod,  M.  J.  Merritt.  Characteristics  of 
Software  Quality.  TRW  Software  Series  TRW-SS- 
73-09,  December  1973 . 

17.  Thayer,  T.  A.,  et  al.  Software  Reliability  Study^ 

(Final  Technical  Report  No.  76- 

2260.1.9-5,  lS7S. 

18.  Llskov,  8.  H.,  and  S.  N.  Zllles.  “Prograafting  with 

Abstract  Data  Types,"  ACM  SIGPLAN  Notices.  April 
1974,  S0-S9. 

19.  Stepcjyk.  F.  M..  Requirements  for  Secure  Operating 

Systems.  TRW  Software  Senes,  rRH-35-7<-05,  June 
ISTT 


Boehm,  3.  W..  R.  K.  McClean,  and  0.  3.  Urfrig, 
"Soaft  Experiences  with  Automateo  Aics  to  the  De- 
sign of  Large-Scale  Software.'  IEEE 
ware  Engineering,  March  1973,  pp.  ...s-l'jj'i 


21.  Teicnroew,  0.  ana  H.  Sayarl,  “Automation  of  System 

Building,"  Oatamacl on.  August  1971,  pp.  25-30. 

22.  Alford,  M.  W..  Jr.,  "A  Reoulrements  Engineering 

Methodology  for  Real-Time  Prccessir.g  Seouirements. " 
Proeeedlrws,  IEEE-acm  Secono  Interraticnai  Confer- 
ence on  Sof^are  Engineering,  uctecer  1976. 

23-  Sell,  r.  E..  and  Q.  C.  Sixler.  "An  Extendable  Ap- 
oroadr  to  Computer-Aided  Software  Reoulrements 
Engineering,"  Ibid. 

24.  Mills,  H.  0.,  Mathematical  Foundations  of  Stw- 

tured  Prooreeunq.  iBH-fSO  Report  72-6012.  197Z. 

25.  9rown,  J.  R.,  P-oceedlnos  of  the  AHE  Conferwee 

on  Software.  Wasnington,  a.t.,  July  19-21,  1976. 

26.  Fagen,  M.  E.,  Design  and  CoO^  Inspections  and 

Procoss  Control  in  the  Deveiooment  of  Programs. 

IBl-  ni-zi-572.  oecemoer  197i7 


Aooendlx 

Definitions  of  Quallc/  Characteristics 

ACCESSIBILITT;  Code  oossesses  the  characteristic 
jisswrilnlttb  to  the  extent  that  It  facilitates  se'se- 
t'.e  use  of  'its  parts.  (Examples:  varlaole  dimens'oneo 
amys,  or  not  using  aosoiute  constants.)  A«je*ss-:Ei itsv 
!s  necessary  for  tfficiwney.  seewoilisvj  nuxwn 
,-rgzn»0rinf, 

ACCOURTABILITT:  Codi  possesses  the  characteristic 
iiaeoioitobiUey  to  the  extant  that  its  usage  can  oe 
iwasured.  t 

This  weens  that  critical  segments  of  code  can  oe  • 
Instnmiented  with  probes  to  measure  timing,  wnether 
specified  bnnehes  are  exercised,  etc.  Cooe  used  “or  ^ 
proocs  Is  preferably  invoked  by  conditional  assemoly 
teennioues  to  eliminate  the  aoditlonal  Instruction  words  | 
or  added  execution  times  wnen  the  measurements  are  not 


ACCURACY;  Code  possesses  the  characteristic  sj- 
purocy  to  Che  extent  that  its  outputs  are  sufficiently 
precise  to  satisfy  their  Intended  use.  Necessary  for 
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rtliabilitV' 


AUSWUTABILITY:  Code  possassts  tht'  charactarts- 
tic augimmtability  to  tha  axtant  tuat  It  can  easily 
accoandata  expansion  In  component  coeoutatlonal  fun— 
tions  'rdat*  storage- raqulremants.  This  It  a neces- 
sary characteristic  for- modi fiabilitH' 

COWUhICATIVENESS:  Coda  possasset  tha  character- 
istic oannattoaeivanaaa  to  tha  extent  that  it  faclll- 
tatat  the- specification- of  Inputs  and.  provides  outputs. 
Mhosa  for*  and  content  ara  easy  to  isslnllata  and  usa- 
flil.  CSoNBattccttivanaea  Is  nacassary  for  uatabiliey 
and  hunm  mtgimmorimg. 

COHPLnEKESSc  Coda  possesses  tha  cnaractari  Stic 
oeopUtomM*  to  tha  extant  that  all  Its  parts  ara 
present  and.  each  part  Is  fully  davalopad. 

This  Implies  that  extamal  rafercncas  ara  avalT- 
ahla  and  required  functions  ara  coded  and  present  as 
designed,,  etc, 

CONCISENESS^  Coda  possesses  tha  cnaractari  Stic 
donoiaaMsa  to-  tha  extent  that  excess  Iva  Informatlon- 
1s  not  present. 

This  Implies  that  prograns  ara  not  excessively 
fiaipantiil  Into  modules,  overlays,  functions  and  suo- 
, routines,  nor  that  tha  sane  seouenca  of  coda  Is  re- 
peated In  nuaerous  places,  rather  than-daflnlng^  a 
subroutina  or  macro;  etc, 

CONSISTENCT:  Coda  possesses  tha  charactaHstIc 
■ixtmemt  eeniotmnay  to  tha  extent  that  It  contains 
unifbm  noUtlon.  terminology  ana.  symbology  within  It- 
self, and.ecPems£  ooneietaney  to  the  extwtt  that  tha 
content  Is  traceebla  ta  tha  requirements. 

tntamal  consistency  Imlles  that  coding  styi- 
derds  are  homogeneously  adhered-  to;  a.  7..  coamants 
should  not  ba  unnecessarily  extensive  or  *«rdy  at  one 
place,  and  Insufficiently  Infometiva  at  another,  tnat 
nuborOf  argtaents  In  subroutina  calls  match  with 
subroutine  header,  etc.  External  consistency  Implies 
that  variable  names  and  definitions.  Including  physi- 
cal units,  ara  consistent  with  a Slossary;  or.  there 
Is  a ona-one  relationship  between- functional  flow 
chert  entitles  and  coded  routines  or  mooules,  ate. 

OEVICE-tNOCPENOENCE.*  Coda  possesses  tha  charac- 
teristic ifaii^i'e  1’ III fepani feme  to  the  extant  It  can  bn 
execuM  on  computer  hardware  configurations  other 
than  Its  current  one.  Clearly  this  characteristic 
Is  a nacassary  condition  for  poroeWlity. 

EFFICIENCY:  Coda  possasses-  tha  cnaractari Stic 
to  the  extent  that  It  fulfills  Its  purpose 
J without  waste  of  resources. 

This  InpTles  that  choices  of  source  code  coir- 
structlons  are  made  In  order  to  produce  the  irlnliw 
maiber  of  words  of  object  code,  or  that  where  alter- 
nate algorithms  are  available,  those  taxing  the  least 
time  are  chosen;  or  that  InfOrmatlon-oacklng  oensity 
In  core  Is  high,  etc.  Of  course,  many  of  the  ways  of 
coding  efficiently  are  not  necessarily  efficient  In 
tha  sense  of  being  cost-effective,  since  portability, 
maintainability,  etc.,  may  be  degradeo  as  a result. 

MjMM  EMINEENINSr  Coda  possesses  the  character- 
istic lemam  myimerwip  to  the  extent  tnat  It  fulfills 
Its  purpose  without  wasting  the  users'  time  and  energy, 
or  degrading  their  morale.  This  characteristic  1m- 
j piles  moaoMihility,  robuetneM,  and  aammiaaSvM- 

; I ■ 
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t 

LESUILITY:  Coda  possesses  tha  characteristic 
to  the  extent  that  Its  function  Is  easily 
discerned  by  reading  the  cooe.  (Examole:  complex  ex- 
pressions have  mnemonic  variable  names  and  parentneses 
even  If  unnecessary.)  is  necessary  for  un- 

iareto^obt  liPy. 

MINTXINA8n.ITr:  Coda  possesses  the  characteris- 
tic maineatnabilitv  to  tha  extent  chat  it  facilitates 
updating  to  satisfy  new  requirements  or  to  correct 
deficiencies. 

This  Implies  that  the  coda  is  understandable, 
castable  and  modIflabU:  e.g. , coRMnts  are  used  to  lo- 
cate subroutina  calls  and.  entry  oolnts.  visual  search 
for-  locations  of  branening  stataments  and  their  targets 
is  facilitated  by  special-  formats,  or  tna program  Is 
designed  to  fit  into  avallabla  resources  with  plenty  of 
margins  to  avoid  major  redesign-,  etc. 

mOIFlASILITY:  Code  possesses  tha  cheractarlstic 
•vdifiabilit^  to  tha  extent  thit  It  facilitates  tha  In- 
corporation of  changes,  onca  tha  nature  of  the  desired 
chenga  hes  been- determined,  'iote  tne  nigner  level  of 
abstractness  of  this  characteristic  as  compared  with 
xu^mntabilisy. 

PORTABILITY:  Code  possesses  tna  cheracter-lstic 
portability  to  tha  extent  chet  1 1 cur  oa  operaiad 
easily  and  well  on  computer  configurations  otner  cnair 
i ts  current  one-. 

This  Imlles  that  special  languega  features,  not 
easily  available  at  other  facilities,  ara- not  usad;  or- 
that  standard  1 ibrary  functions  ana  suoroutinas  ara  se- 
lected for- universal  applicability,  ate. 

RELIABILITY:  Codr  possesses  tha  charactaristlc 
rmlu±ility  to  the  extent  thet  it  can  br  axpected  to 
perform  Its  Intended  functions  setlsfectortly. 

This  liiBlIes  thet  tha  progrw  will  comolle,  load, 
and  execute,  producing  answers  of  tna  requisita  accur- 
acy; and  that  tha  program  will  centinua  to  oparata  cor- 
rectly. except  for  a tolerably  small  ntaParof  in- 
stances, wn1le  In  operational  usa.  It  also  Implies  that 
It  Is  comolcta  and  axtamally  consistent,  ate. 

ROBUSTNESS:  Coda  possassts  tha  charactaristlc 
fobuotmmoo  to  tha  axtent  that  it  can-  continue  to  perform 
despite  soma  violation  of  tha  assiaptlons  in  its  speci- 
fication. 

This  Imollas,  fbr  axampla.  thac  tha  program  will  < 

properly  hanola  Inputs  out  of  range,  or  in  different 
format  or  type  than  dafinad.  wltnout  dagraolng  its  oar-  | 

formanca  of  functions  not  dependant  on  the  non-standard  j 
inputs. 

SELr-CONTAlNEDNESS:  Code  possesses  the  therecter-  ^ 
Istic  aw tf-o9Ktaim*dntnt  to  tha  extent  thet  It  oerfonis  f 
all  Its  explicit  and  liwllclt  functions  within  itself.  } 

Exasples  of  Imolldt  functions  ere  Inltlellzetlon,  in- 
put checxlng.  diagnostics,  ate. 

SaF’-OESCRIPTIVENESS : Coda  oossessas  the  cherae- 
arlstlc  Mlf-dMorittittnott  to  the  extent  thet  It  con- 
tains enough  Information  for  a reader  to  determine  or 
verify  its  objectives,  assurations,  constraints.  Incuts, 
outputs,  components,  ana  revision  status.  Conrentary 
and  traceability  of  previous  cnanges  by  transforming 
previous  versions  of  code  Into  non -executable  but  pres- 
ent fir  avellable  by  mecro  calls)  code  are  some  of  the 
weys  of  providing  this  cnaracterlsdc.  Solf-dMtrip~ 
pitMweee  Is  necessery  for  both  ttnability  and  •Mrdtrr- 
ttmrdbbility. 
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rntUCTUREONESS:  Code  oosscsscs  tht  characttrls* 
Me  »tructur*tiuj»  to  tne  extent  tnat  it  sossesses  a 
daMnite  pattern  of  organization  of  its  interdepen- 
dent parts. 

This  implies  that  evolution  of  the  oroqram  de- 
sign has  proceeded  in  an  orderly  and  systentatic  manner, 
and  that  standard  control  structures  have  been  followed 
in  coding  the  program,  etc. 

TESTABILITY;  Code  possesses  the  characteristic 
uatability  to  the  extant  that  it  facilitates  the  es- 
Ublishnent  of  verification  criteria  and  supports  eval- 
uaMon  of  its  performance. 

This  Implies  that  repuirementr  are  matched  to  spe- 
cific nodules,  or  diagnostic  capaoilities  are  provided, 
etc. 


UNOERSTANQABILITY:  Code  possesses  the  character- 
IsMc  wtdMTttaiviBbility  to  the  extent  that  its  purpose 
is  clear  to  the  inspector. 

This  implies  that  variable  names  or  symbols  aro- 
used consistently,  modules  of  code  are  sel f-descrip- 
Mve,  and  the  control  structure  is  simple  or  in  ac- 
cordance with  a prescribed  standard,  etc. 

USABILITY  (AS- IS  UTILITY):  Code  possesses  the 
characteristic  usability  to  the  extent  that  it  is  re- 
liable, efficient  and  numan-engineered. 

This  implies  that  the  function  performed  by  the- 
progm  is  useful  elsewhere,  is  rooust  against  human 
errors  (e.g.,  accepts  either  integer  or  real  reoresen- 
tttlons  for  type  real  variables),  or  does  not  require 
excessive  core  memory,  etc. 
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MAMACEMENT  OF  SOFTWARE  DEVELOPMENT  * 
Edmund  B.  Only 

GTE  Aucomncic  Electric  Laboratories 


ABSTRACT 


This  paper  describes  five  major  aspects  of 
software  management:  Development  Statistics, 
Development  Process,  Development  Objectives,  Orga- 
nization Structure  and  Software  Maintenance.  The 
control  of  both  large  and  small  software  projects 
is  included  in  the  analysis. 
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The  concepts  presented  in  this  paper 
have  been  derived  from  the  management  tech- 
niques that  were  employed  in  three  large 
sottware  development  projects.  These 
projects  accumulated  nearly  2,000,000  hours 
of  software  development  experience  and 
required  t.he  generation  of  large  real-time 
programs,  extensive  supporting  programs  and 
small  minicomputer /microcomputer  real-time 
programs . 

The  software  projects  completed 
development  at  three  successive  points  in 
time.  The  latest  project,  capitalizing  on 
experience  gained  during  the  rvo  earlier 
projects,  was  developed  exactly  on  schedule 
and  10%  under  budget.  All  available 
statistics  indicate  that  the  third  project 
was  a more  efficiently  managed  project  than 
the  earlier  two. 

Although  the  data  in  this  paper  is 
historical,  some  of  the  proposed  management 
concepts  have  not  been  used  extensively,  but 
are  methods  established  to  overcome  earlier, 
leas  acceptable,  management  practices. 

This  paper  is  based  on  the  following 
premise: 

- Without  a planned  development 
methodology,  software  management  will 
fail. 

- Without  an  organization  structure 
tailored  to  the  selected  development' 
methodology,  software  management  wi..,. 
fail. 

- Without  achievable  obj  actives , formal 
controls  and  realistic  estimating 
techniques  software  management  will 
fail. 

DEVELOPMEirr  STATISTICS 

The  unknown  elements  of  software 
management  can  be  reduced  by  employing 
historical  development  statistics  as  guide- 


lines for  estimating  and  controlling  current 
development.  Often  these  statistical  guide- 
lines are  used  by  management  to  establish 
development  objectives  without  spending 
sufficient  time  to  investigate  the  foundation 
upon  which  the  data  is  generated. 

The  Total  Software  Job 

This  section  looks  at  the  software 
development  process  from  three  new  points: 
first,  the  total  software  job;  second,  the 
components  of  a software  job;  third,  the 
dynamic  flow  of  a software  job. 

GTE  AECo . has  developed  and  cut  into 
service  three  large  stored  program  controlled 
telephone  switching  machines.  The  first 
project  completed  software  integration  in 
1972  and  Incurred  a development  rata  for  on- 
line software  of  3.0  hours  per  machine 
instruction.  This  program  contained  160,000 
machine  instructions.^  The  second  and  third 
projects  completed  development  in  early  1973 
and  late  1974.  They  encountered  development 
rates  of  2.3  hours  per  machine  instruction 
and  1.9  hours  per  newly  developed  machine 
Instruction,  respectively.  Each  of  these 
latter  two  projects  were  manned  by  completely 
different  prograoraers,  supervisors,  and  unit 
heads.  Both  of  these  latter  projects  re- 
quired approximately  112,000  new  machine 
inatruccions  to  be  developed,  tn  the  above 
eases,  development  hours  include  all  efforts 
required  for  specification,  design,  code, 
test  and  maintenance  up  eo  coBSsercial 
availability  of  the  program. 

When  looking  at  these  statistics  one 
may 'ask  the  following  question;  Is  this 
development  rate  good  or  bad?  The  answer 
could  be  "bad"  because  software  development 
rates  can  easily  reach  six  minutes  per  source 
line;  but  then,  the  answer  could  al.xo  be 
"good"  because  some  managers  feel  that 
crying  to  develop  a program  requiring  more 
chan, 30  programmers  is  close  to  impossible. 

In  order  to  compare  the  development  races 
of  rwo  different  programs  the  following 
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factors  tnusc  be  considered: 


Ocher  Factors  That  Affect  JeveLoor.er.t  Rate 


- Design  objectives 

- Program  size 

- Program  complexity 

- Program  language 

- Program  environment 

- Data  base 


In  addition  to  crying  to  optimize  all 
objectives,  the  400  to  1000  source  lines  per 
man  year  program  usually  encounters  other 
manpower  devouring  reouirements , the  effort 
for  which  is  usually  buried  into  the  devel- 
opment rata.  These  factors  include; 

a.  Large  Data  Base  Pecuirements  -The 
development  effort  required  to 
structure,  define  and  document  the 
data  base. 


- Documentation  standards 

- Personnel  and  computer  resources 
Design  Objectives  .if fact  Dave Looment  Pates 

A program  can  be  designed  to  optimize 
one  or  more  of  the  following  standards: 

Memory  Utilization  - Perform  the 
required  functions  using  the  least  amount  of 
memory  space  to  house  both  instructions  and 
data  base.  This  objective  attempts  to 
minimize  system  memory  costs. 

Executive  Speed  - Perform  the  required 
functions  using  mini-mum  real  time  during 
execution.  This  objective  attempts  to 
maximize  system  through-put. 

Schedule  - Perform  the  desired 
functions  in  t.he  least  am.ounc  of  ti-me.  In 
a competitive  market  an  "excellent”  late 
product  may  be  less  profitable  than  an 
’acceptable"  early  product. 

System  Maintenance  - Perform  the 
desired  functions  so  as  to  minimize  on-sita 
maintenance  cost. 

High  Quality  - Insure  that  the  required 
functions  are  performed  by  well  documented 
and  thorcughlv  tested  software.  Design 
cannot  rely  on  the  customer  to  find  t.he  last 
S'  of  sofrware  bugs. 

Design  Cost  - Perform  the  required 
functions  with  minimum  e.xpenditure  of 
resource  dollars.  This  expenditure  is  in 
terms  of  manpower,  material  and  travel. 

This  objective  should  include  both  initial 
design  as  well  as  design  maintenance. 

Establishing  quantitative  values  for 
these  objectives  is  not  a simple  matter  - at 
least  not  for  the  programs  which  are 
developed  at  a rate  of  400  to  1000  source 
lines  per  man  year.  Unlike  the  programs 
which  are  developed  at  higher  rates,  these 
400  to  1000  source  lines  per  mar.  year 
programs  attempt  to  simultaneously  optimize 
all  the  above  objectives.  To  use  an  extreme 
exat&plc:  One  month  the  pressure  is  aimed 
at  reducing  memory  size  in  order  to  meet 
competitive  manufacturing  costs.  The 
following  month  generates  pressures  to 
either  reduce  development  dollars  due  to 
smaller  budgets  or  improve  real  time  in 
order  to  capture  a larger  market. 


b.  Large  and  Complex  Program  - 
Requiring  from  75. DOC  source  lines 
to  500,000  source  lines.  These 
programs  must  pay  the  cost  to 
integrate  the  output  of  many 
programming  teams . 

c.  Extensive  Commercial  Documentaci'on 
- Such  as  diagnostic  dictionaries, 
inpuc/outpuc  message  manuals,  user 
documents,  charts,  descriptions 
and  extensively  commented  listings. 

d.  Program  Specified  by  An  External 
Source  - The  design  team  does  not 
have  the  option  to  decide  what  , 
feature  requirements  the  program 
will  satisfy  - as  such,  c.hesc 
requirements  are  often  loosely 
defined  and  constantly  change 
during  the  early  period  of  develop- 
ment . 

e.  Generic  Structure  - .A  single 
program  must  be  able  to  work  in 
a»a.tiy  different  envirorjaents  - the 
data  structure  must  be  designed  to  - 
cope  with  varying  requirements . 

Thus,  the  program  remains  constant 
and  the  data  base  information 

. content  and  size  changes  from  sice 
CO  sice. 

f.  Complex  Interfaces  - Proeram  must 
be  able  to  work  with  different 
quanctcles  of  hardware.  The 
program  also  has  e.xcenslve  hardware 
/software  interfaces  as  well  as 
man/machine  interfaces. 

g.  Sofrware  Responsible  for  System 
Reliability  - Sofrware  controls 
are  designed  to  minimize  the  affect 
on  system  operation  due  to  hardware 
faults,  hardware  noise,  data  base 
errors  or  instruction  errors. 

h.  Maincelnable  • Programs  are 
designed  to  last  twenty  years  with 
active  maintenance  occurring  at 
least  during  the  first  four'years. 
Good  structured  design  is  thus 
required. 

Resources  Also  Affect  Develoomcnc  Rates 

Real-time  software  development  races 
ere  vary  dependent  on  the  resources 
available  to  menagement.  Three  important 
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- The  prograoner 

- Support  software  (we  will  discuss  only 
two  support  packages;  coopller  and 
simulator) . 

- Support  computer. 

Prograoner  - A Resource  - In  most 
applications  a balanced  team  consisting  of 
one  leader,  two  experienced  progranmers  and 
three  Inexperienced  energetic  prograosners  Is 
Ideal  during  the  laplementatlon  and 
maintenance  phases . A smaller  team  of 
experienced  system  level  programmers  Is 
usually  required  during  the  earlier  planning 
and  specification  phases. 

Compiler  - A Resource  - A good  compiler 
allows  programmers  to  code  In  hlgh*level 
language  without  extensive  waste  of  memory. 

A 20"  reduction  In  total  development  effort 
has  been  achieved  at  GTE  .AECo.  when  a real- 
time program  was  designed  using  high-level 
language  rather  than  assembly  language.  This 
reduction  In  effort  occurs  In  the  following 
design  phases;  coding,  documentation, 
testing,  design  maintenance.  Because  a 
source  line  written  in  high-level  language 
contains  more  Intelligence  than  does  a source 
line  written  In  assembly  level  language,  a 
high-level  language  source  line  often  Incurs 
a larger  "development  cost  per  source  line" 
than  does  the  siller  assembly  level  source 
line.  An  assembly  level  source  line  usually 
produces  one  machine  Instruction.  A high- 
level  source  line  usually  produces  more  than 
one  machine  Instruction. 

Simulator  - A Resource  - Large  real- 
time programs  must  be  eventually  tested  on 
Che  machine  In  which  they  will  reside.  This 
mechine  Is  called  the  "object  machine".  The  ' 
amount  of  object  machine  time  required  to 
test  a program  depends  heavily  on  the  amount 
of  prior  program  testing  which  can  be 
performed  off-line  as  well  as  the  quality  of 
the  object  machine  utilities.  One  method  of 
off-line  testing  is  called  simulation.  The 
process  requires  the  design  of  a support 
program  called  a simulator.  This  simulator 
makes  an  off-line  computer  look  like  the 
object  machine;  at  least  to  the  real-time 
program  which  Is  being  tested. 

For  large  real-time  programs,  such  as 
Che  ones  described  eerller,  one  hour  of 
object  machine  time  Is  required  to  test  ten 
Instructions  of  unslmulaced,  asse^ly  level 
code.  If  simulation  Is  used  prior  to  object 
machine  testing,  this  race  Increases  to 
twenty- five  instructions  pec  hour.  The  least 
aanunc  of  object  machine  time  Is  required  for 
simulated  programs  which  were  coded  In  high- 
level  language.  These  programs  have  been 
tested  on  Che  object  machine  at  an  average 
race  of  forty  Instructions  per  hour. 

The  reduction  In  amount  of  machine  time 
required  for  testing  affects  development 
expenditures  In  two  ways; 


a.  A programmer  reculres  three  hou -s 

of  preparation  and  analysis  time  for 
each  one  hour  spent  testing  on  the 
object  machine.  Therefore,  a one 
hour  reduction  In  machine  time 
carries  a four  hour  reduction  In 
development  hours . 

b.  Object  machine  time  Is  very 
expensive.  For  the  large  real-time 
programs  described  In  this  paper 
machine  costs  range  from  $40.00 

to  $130.00  per  hour. 

Computer  - A Resource  - In  addition  to 
supporting  software,  the  quality  and  quantity 
of  computer  resources  affects  efficiency  of 
coding  and  testing.  Currently,  Interactive 
editing  and -testing  seems  to  be  the  most 
effective  way  to  debug  spftwere.  Batch,  how- 
ever, Is  still  an  effective  development  tool 
as  long  as  turn-arovind  time  remains  under 
2 . 5 hours . 

In  conclusion,  development  races  should 
only  be  used  as  a guideline  for  planning  new 
designs.  These-rates  cannot  easily  be 
compared  - even  for  similar  jobs  - since 
management  objectives  or  available  resources 
may  be  different.  In  any  case,  the  program 
Incurring  the  more  costly  development  rate 
may  be  the  better  design  approach  since  It 
could  require  less  memory  and  execute  faster, 
thus  requiring  a less  costly  memory  system 
and  forcing  a larger  market.  The  object  of 
sofrware  management  should  always  be  to 
optimize  profits.  This  objective  may  require 
larger  Initial  development  expenditures  In 
terms  of  hours  per  source  line. 

Different  Program  Classifications 

With  this  background,* let  us  now  look  ' 
at  the  development  race  for  various  types 
of  programs  In  order  to  get  some,  feeling  of 
how  development  races  fluctuate.  The  range 
of  development  varies  from  0.24  machine 
In.scrucclons  per  hour  to  12  machine 
instructions  per  hour.  I do  not  believe  that 
Che  high  development  rate  Indicates  better 
management  but  merely  a different  software 
job. 

For  simplicity  let  us  categorize 
programs  Into  three  areas;  Small  Real-Time, 
Large  Real-Time,  Support. 

Small  Real-Time  Programs 

Data  has  been  compiled  from  five 
different  minicomputer  developments.  These 
programs  usually  range  from  5,000  to  20,000 
machine  Instructions  and  range  of  effort 
varies  from  1.6  machine  inacruccions  per  hour 
to  5 machine  Instructions  par  hour.  Inter- 
medlete  rates  are  2.3  machine  Instructions 
per  hour  and  1.9  machine  Inacruccions  per 
hour. 

The  program  which  was  developed  at  5 
mechine  instructions  per  hour  did  not  require 
coimnerclal  documentation,  nor  was  memory 
usage  or  real  time  a strong  guideline. 
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Managerial  objectives  vere  to  generate 
prograa  in  ainiaal  tiae,  reouiriag  ainiaal 
aan/aachine  interface.  The  systea  was 
designed  to  drive  tiaffic  through  a larger 
aachine.  Systea  up-tiae  was  not  an  iaportant 
factor.  The  prograa  was  designed  by  one 
experienced  prograaaer.  The  software  was 
operational  in  three  aonths. 

The  programs  which  were  developed  at  a 
slower  rate  faced  completely  different 
design  objectives  and  environments.  They 
required  large  data  base  structures.  Real- 
tiae  execution  as  well  as  system  up-ti.ae  were 
iaportant  factors.  These  systems  required 
cotzercial  documentation  and  were  designed 
for  low  cost  software  maintenance  (i.e., 
nodular  constr-jction)  . 

Large  Real  Tiae  Programs 

Chart  1 shows  development  statistics  for 
three  large  real-time  programs.  .^11  three 
programs  were  designed  to  meet  the  most 
stringent  operational  requirements.  The  most 
important  single  factor  which  determined  the 
different  development  rates  is  the  'experience 
level  of  the  designing  programmers  and 
supeir/isors . 


CHART  1 


SYSTEM 
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instructionsj 
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.43  1 

c 

19T4 

1 11.000 
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The  above  programs  are  constructed 
using  many  subprograms.  On  an  average  each 
subprogram  houses  3,000  machine  instructions 
or  20  software  modules.  The  development  rate 
for  subprograms  is  different  chan  for  the 
total  program  in  that  some  subprograms  are 
very  complex-  such  as  diagnostic  suborograms 
- and  some  are  rather  simole  - pure  data 
manipulation.  The  following  list  gives  rwo 
examples  of  subprogram  development  races. 

a.  Data  Manipulation  - such  as  updating 
saiaory  information.  1.9  aachine 
instructions  per  hour  for  sub- 
programs . 

b.  Diagnostic  - such  as  routining 
electronic  hardware  to  determine  if 
it  is  operational.  0.24  machine 
instructions  per  hour  for  sub- 
programs . 

In  each  case  an  additional  effort  of  67. 
is  required  to  integrate  each  subprogram  into 
a total  program. 

Support  Programs 

These  programs  are  intermediate  size 
(20,000  source  lines).  The  programs  referred 


CO  in  this  section  are  called  support  software 
because  they  have  been  designed  to  "support" 
the  design  and  manufacture  of  the  large  real- 
ti.me  programs  described  in  the  last  section. 
Support  programs  consist  of  loaders,  editors, 
assemblers,  compilers,  program  generators, 
simulators  and  utility  programs.  They  have 
been  designed  to  execute  on  I3.M-360/  370 
machines.  The  size  of  support  software  is 
large:  400.000  source  lines  of  support  soft- 
ware has  been  designed  to  "support"  the  design 
and  manufacture  of  3'0,OOC  m.ac.hine  instruc- 
tions of  real-time  software.  Support 
programs,  however,  are  '.ess  complex  to  design 
than  real  time  operational  orozrams . Also. 

60".  of  the  code  is  im.plemenced  using  high- 
level  language.  The  development  rate  ranges 
from  0.7  source  lines  per  hour  for  the  more 
complex  support  programs  to  12  source  lines 
per  hour  for  the' simpler  programs.  The 
average  developr.ent  race  for  the  entire 
package  of  400.000  source  lines  is  1.3  source 
lines  per  hour. 


^^oc  only  is  the  initial  development 
effort  associated  with  support  software  less 
(per  source  line)  than  that  required  for  large 
reel  time  programs  but  the  design  maintenance 
effort  is  also  less.  Statistics  based  on 
maintaining  three  separate  real  time  software 
packages  and  two  separate  support  packages 
indicate  the  following  design  maintenance 
effort ; 

a.  One  equivalent  programmer  (two 
programmers  - each  devoting  half 
time  CO  maintaining  software)  can 
maintain  from  15.000  to  30,000  source 
lines  of  on-line  r«el-cime  programs. 
This  can  be  co.mpared  to  the  rate  for 
support  software  of  from  30,000  to 

120.000  source  lines  per'  equivalent 
programmer . 

b.  One  programmer,  devoting  full  tine  to 
maintaining  software,  can  maintain 

10.000  source  lines  of  on-line  real 
time  programs  - or  30.000  source 
lines  of  support  programs . This 
number  differs  from  that  presented 
in  "a"  in  that  we  have  found  that 
design  maintenance  progracmiers  are 
more  efficiently  employed  if 
approximately  50*  of  their  working 
hours  are  spent  in  new  design. 

The  maintenance  races  given  above  do  not 
include  such  overhead  items  as  configuration 
control,  laboratory  and  field  support, 
supervision. 

Develoomenc  Races  - Comoonents 

In  addition  to  specifying  development 
hours  per  source  line,  another  important 
development  statistic  divides  this  total 
deve''^''fflenc  effort  into  its  component  parts: 
spec  . kicacion,  design  (Includes  code) , test, 
documentation,  design  maintenance. 

Chert  2 is  a pie  chart  which  shows  this 
division  for  the  three  large  real-time 
programs  described  earlier!  Although, 


Develooment  Rate.-  Design  Maintenance 
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^67,  of  coeal  sofr<rar«  d«v«Iopmenc  dollar 
expandicurt  ac  a period  of  i*  /ears  after 
coenerclal  avallablllc/.  Since  the  aajor 
aalncenance  effort  takes  place  durln?  the 
first  four  years  after  the  comerclal  release 
of  a prograa,  the  ^67,  should  Increase  very 
slowly  - possibly  approaching  607.  after  10 
years.  Design  ir.alnter.ance  as  used  In  Chart  i* 
includes  laboratory  efforts  associated  with 
fixing  design  problems  and  with  small  program 
enhancements.  It  does  not  Include  efforts 
required  for  large  feature  additions  or  local 
"on-slte"  support. 


Chart  2- represents  an  average  of  the  three 
p'rograms . each  program  taken  Individually 
Indicates  a similar  distribution.  However. 
Chart  2 does  not  include  the  effort  required 
to  "evaluate"  the  software  package.  This 
effort  usually  takes  place  at  the  first  field 
site.  For  larze  real-time  programs 
approximately  0.1  hours  per  instruction  is 
required  to  write  and  execute  evaluation 
test  plans.  The  maintenance  effort  shown  in 
Chart  2 is  used  to  resolve  sofrware  bugs 
found  during  evaluation. 


CHART  2 
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Chart  3 shows  the  same  breakdown  as 
Chart  2 but  at  a different  point  In  time. 
Note  that  design  maintenance  effort  has 
Increased  from  12^  of  total  effort  (one  year 
after  In  service) , to  of  total  effort 
after  four  years  of  service. 


Dynamic  Flow 


The  software  development  statistics 
discussed  to  this  point  have  been  static 
staclatlcs  - taken  at  one  point  in  time. 
Charts  S and  6 show  how  a Software  project  ' 
progresses  through  its  various  phases.  These 
charts  Illustrate  statistics  for.  two 
different  large  real-time  programs  - devel- 
oped by  two  different  teams  of  programmers, 
supervisors  and  unit  heads. 

Chert  S shows  how  estimated  program  size 
varies  throughout  the  period  of  development. 
"Estimated  program  size"  Is  a projection  of 
the  program  size  at  commercial  availability. 

A surprising  "characteristic  curve"  is 
obvious  from  this  chart.  In  both  projects, 
program  size  estimates  made  early  in 
development  (start  of  specification)  were 
high  and  then  rapidly  dropped  - hitting  a 
low  abb  at  the  start  of  desl^.  Estimated 
program  size  than  starts  to  Increase  ac  a 
conservatively  slow  race  until  design  reaches 
60%  complete.  Ac  this  point  In  time  both 
projects  Increased  estimated  program  size 
estimates  at  a very  rapid  race  - resulting 
In  a size  Increase  of  about  30%  from  point  A 
CO  point  C.  After  design  completion  reached 
the  98%,  program  size  estimates  level  off  and 
were  within  5%  of  the  actual  program  size  ac 
coniBarclal  availability. 

Chart  6 Is  an  actual  plot  tracing  the 
development  of  the  110,000  Instruction  real- 
time program,  we  see  the  progress  of  software 
development  for  specification,  design, 
integration  and  evaluation.  Mote  that  In  all 
cases  there  Is  a natural  overlap  of 


CHART  3 

OEVB-OPMeqT  PIE  CHART  INCLUDING  DESIGN 
MAINTENANCE  TO  FOUR  YEARS  AFTER  IN  SERVICE 


WITHOUT  OVERHEAD 


Charts  2 and  3 Include  design  mainte- 
nance effort  only  to  the  extent  chat  it  Is 
performed  by  a prograsner.  This  effort  is 
only  33%  of  the  total  expenditure  allocable 
to  meincelnlng  real-time  software.  The 
remaining  43%  of  total  maintenance  cost 
arises  from  prototype  and  field  support  man- 
power which  Is  used  to  support  the  program- 
mers testing  activities,  to  evaluate  his 
tested  programs  and  to  place  these  programs 
under  configuration  control.  If  we  allocate 
these  overhead  charges  to  design  maintenance 
Chart  4 results.  This  chart  indicates  that 
design  maintenance  of  sofrware  accounts  for 


acclvlries , in  thac  ac  any  one  ooinc  in 
sany  phases  of  develoor.ent  are  raking  p 
siaulraneousiy . Charr  6 shows  char  the 
required  zo  ooaplece  the  lasc  20",  of  a 
opaen:  phase  is  as  long  as  :he  design  : 
coapLere  che  aiddle  "O"..  This  characra 
is  responsible  for  nany  schedule  slippa 
since  a good  pore  ion  of  the  acrivirv  be 
perioraed  during  che  larter  20"  cons  is; 
correcting  r.iscakes:  either  aver.ooked 
aodules  or  unaccepcable  design. 
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CHART  6 

CHARACTERISTIC  'S'  CURVE 
FOR  SOFTWARE  DEVELOPMENT 


WEEKS  INTO  PROJECT 


Deve lopmenc  Scatiscics  - A Tucure  Trend 


Although  this  section  is  concerned  with 
software  develcpeent  rates  - we  can  crack 
Che  future  evolution  of  these  races  by 
comparing  them  to  chose  being  experienced  in 
Che  related  field  of  electronic  hardware. 


develop  one  source  line  of  software 
than  to  develop  one  hardware  logic 


b.  The  sa.r.e  anount  of  effort  is  required 
to  develop  a large  functional  soft- 
ware segment  as  a functional  logic 
board  in  hardware  (containing  30 
gates)  . 

c.  Using  the  total  number  of  system 
source  lines  (160. OCC)  and  total 
number  of  logic  gates  (1*0,000)  as 
a base  for  comparison,  then  design 
maintenance  correcciSns  per  source 
line  exceed  design  maintenance 
ccrrecctons  per  logic  gate  by  a 
factor  of  four  to  one.  Design 
maintenance  cost  for  software  also 
exceeds  design  maintenance  cost  for 
electronic  hardware  by  a factor  of 
four  to  one . 


d.  System  software  complexity 

approximately  equals  system  hardware 
complexity . 

The  question  which  must  be  answered  is 
"whv  did  total  develocment  cost  per  source 
line  exceed  total  development  cost  per  Logic 
gate  bv  such  a large  factor  when  the  system 
hardware  comple.xity  equaled  the  system  so-ft- 
ware  complexity  and  the  number  of  source 
-lines  equaled  the  number  of  logic  gates?". 

The  answer  lies  in  four  areas: 

a.  Hardware  management  technicues  and 

development  procedures  were  more 
advanced  than  those  used  in  software.  j 

b.  Hardware  designers  were  more 

experienced.  { 

c.  Hardware  employed  a more  "structured 

■design".  ] 

d.  The  basic  building  blocks  used  in  | 

software  design  (i.e.,  different  j 

types  of  source  statements)  were  more 
tnaterous  and  complex  than  the  simple 
building  blocks  used  in  hardware:  . 

AND,  OR.  NOT.  ('Use  of  structured 
progranining  could  eliminate  this 

difference  since  the  technique  limits  I 

software  building  blocks  to  three 

types  of  single-entry,  single-exit  ' 

control  structures:  simple  selection.  I 

simple  repetition  and  linear 

sequence.  Structured  programming 

insures  Chat  a software  source  line 

contains  the  same  amount  of  Intel-  \ 

ligence  as  a hardware  logic  gate.) 


One  of  Che  large  real-time  programs 
referred  to  in  this  paper  contains  160.000 
source  lines  of  software.  This  program 
controls  che  operation  of  a hardware  system 
containing  approximacaly  170.000  logic  gacas . 

Historical  statistics  gathered  in  this 
system  indicate  chat  there  is  a close  analogy 
berween  hardware  and  software  development 
scacifcics.  For  example: 

a.  Twice  as  much  effort  is  required  to 


e.  In  a software/hardware  system  the 
hardware  gets  a double  dose  of 
testing  and  evaluation:  che  first 
dose  occurs  during  schedu.lec  hard- 
ware testing,  the  second  dose  occurs 
as  a natural  byproduct  of  lofrware 
testing.  f-, 

In  conclusion  - as  software  management  ^ 
and  design  techniques  approach  the  level  chat 
exists  in  hardware,  and’ as  software  design 
becomes  .r.or=  structured,  projects  such  as  the 

. J t 
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, on«  described  above  we  can  expect  development 
rates  per  source  line  and  maintenance  costs 
per  source  line  to  decrease  * hopefully 
approaching  races  presently  being  experienced 
by  an  electronic  logic  gate. 

TH£  PROCSSS  OF  SOrTvA.^l-  SE’.^LOPMZMT 

Statistical  data  gathered  during  the 
process  of  software  development  can  be  used 
CO  establish  an  efficient  methodology  for 
future  development.  Tor  example,  historical 
analysis  indicates  that  over  50"  of  all 
development  hours  are  spent  correcting  buzs 
which  result  from  faulty  design.  Information 
such  as  this  tends  to  guide  management  toward 
development  methodologies  which  place  more 
emphasis  in  generating  a higher  quality 
initial  design  with  the  expectation  chat  this 
additional  early  expenditure  will  be  paid 
bacir.  during  testing  and  design  maintenance. 

Since  the  primary  output  of  sofrware  de- 
velopment is  "code",  many  methodologies  place 
heavy  emphasis  on  generating  code  eorly  in 
Che  development  cycle.  These  software  projects 
usually  experience  testing  phases  which  last 
significanclv  longer  than  the  combined  speci- 
fication, design  and  code  phases . Many  of 
Che  projects  which  rush  the  process  of 
generating  code  are  the  same  projects  which 
experience  the  longest  total  development 
schedule.  Chart  6 shows  development 
completion  curves  for  a Large  sofrware  project 
where  generation  of  code  was  not  given  prime 
importance,  even  in  this  project  the  testing 
and  evaluation  phase  lasted  approximately  the 
same  amount  of  time  as  the  combined  speci- 
fication design  and  code  phases. 

Chart  C shows  two  "extreme"  approaches 
to  software  development.  More  experienced 
development  organizations  are  tending  coward 
Method  1 since  it  produces  a higher  quality 
sofrware  product  at  a lower  development  cost 
- espedlally  if  one  considers  long  term 
maintenance . 

Method  2 seems  to  be  a more  natural 
approach  in  chat  it  requires  less  mancgemenc 
control,  requires  less  cechnic.il  (software) 
experience,  allows  orogrataners  to  both  in- 
novate and  avoid  what  they  hate  most  - 
do  cumcn  cation. 

Method  2 will  also  require  Less  memory 
space  and  less  execution  time  than  Method  1. 
The  reasons  for  this  fact  ara  Chraafold: 

> Scructurad  daslgn  raquiras  axcansive 
subrouclna  linkaga.  This  procass  ucl- 
llzas  axcra  mamory  and  exacuclon  clma 
in  ordar  to  maintain  incagrlcy  of 
"computer  register"  information.  (Some 
minicompucars  ara  baing  daslgnad  to 
mlnialza  this  ovarhaad  by  saving  cha 
concancs  of  program  regi'sear  via  hard- 
vara  at  aach  subrouclna  call.) 

- Hlgh-lavel  language  requires  more 
memory  space  and  execution  time  chan 
assembly  level  Implementation  of  the 
same  job.  Even  the  better  compilers 
require  10-15",  more  memory  soace  and 
exacuclon  time.  This  percentage 


increas'es  to  100^  for  many  commercial 
high-level  languages. 

- The  eight  complex  code  employed  in 
Method  2 will  require  less  program, 
storage  space  and  execution  time  than 
the  structured  code  used  in  Method  1. 

A good  guideline  is  to  employ  a metho- 
dology somewhere  between  the  two.  - but  muc.h 
closer  CO  Method  1.  The  three  large  GTE 
projects  referenced  in  this  paper  followed 
this  guideline. 
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A worxable  development  methodology 

The  followlnz  section  descrtbc.s  .i  manage- 
ment approach  which  is  being  used  by  shmc 
newer  projects  in  GTE  AEL  to  eenerace  commer- 
cial software.  The  process  which  is  described 
is  not  a revolution  in  management  techniques, 
but  standard  management  practices  melded  with 
Che  idiosyncrasies  of  software  development. 
These  cechnicues  have  evolved  during  the  de- 
velopment and  maincanance  of  GTE 'a  EAX  and 
TS?S  scored  program  switching  machines. 

There  la  no  universal  process  which 
should  be  used  to  develop  all  types  of  soft- 
ware. Large  commercial  real-time  progrems 
which  involve  more  than  30  programmers  require 
a development  methodology  containing  more 
documentation , more  cross-checks  and  in 
general  - better  management  techniques  chan 
do  smaller  non-cooincreial  programs  chat  may 
only  be  executed  in  a few  machines  and  usually 
by  highly  qualified  engineers.  What  is 
important  is  chat  manaeemenc  establish  a de- 
velopment process  for  each  type  of  software 
design  - earlv  in  the  planning  stage.  The 
methodology  presented  here  is  applicable  to 
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large  raal-rir.e  consercial  sofrvare  packages. 
The  process  can  be  r.Qdibiad  for  "s'-r-pler  " 
deveLccr-enrs  by  eliciinarir.z  scrr.e  oi  the  less 
cricical  dcc-crenrs  and  rechnical  cross-checks. 

Develoor.er.:  Cvcl. 

There  is  a .pacific  cycle  associaced 
wich  che  ger.erarian  af  ccrssercial  safrvare. 
This  cycle  car.  be  divided  inco  five  phases- 
plan,  specify,  design,  cede.  resc.  Alchough 
chese  phases  are  sar.evhac  ur.'.versal  for  all 
sofevare  developnencs . rhe  specific  ctanage- 
senc  aechods  used  ca  insure  successful 
conplecion  af  each  phase  differs  due  ca 
variacior.s  in  nanager.enc  scyles.  and  nore 
iaporrancly . variarions  in  aar.ager.enc 
aacuricy. 

Before  looking  ac  each  of  che  above 
developnenr  phases  in  detail,  there  is  a list 
of  enpiricai  tiar.azer.ent  guidelines  which  seen 
to  be  applicable  in  all  large  software 
developnents . 

a.  Management  review  at  development 

progress  will  not  insure  successful 
completion.  Management  must  e-oect 
not  to  be  told  the  completely  story 
- especially  in  trouble  areas.  Thus, 
detailed  technical  cross-checks  are 


and  will  result  from  a combination 
of  poor  supervision  and  substandard 
programmers . 

These  unacceptable  programs  will  not 
be  detected  by  either  management 
reviews  or  scheduling  technicues 
such  as  ?BR7. 

Management  must  therefore  establish 
techniques  to. find  chese  "bad" 
programs  before  testing  begins  in 
order  to  insure  minimal  effect  on 
project  schedules  and  development 
cost . 

f.  Software  development  costs  can  be 
minimized  by  limiting  the  generation 
of  technical  dccucencacion  to  that 
which  will  be  maintained  for  the 
life  of  Che  software  program  (2Q 
years) . 

Many  develoomenc  projects  ccncr.ntc 
technical  documents  which  irc  not 
maintained.  These  documents  ure  not 
only  expensive  to  generate  but  soon 
become  useless  and  outdated,  or  even 
worse  become  dangerous,  since  they 
contain  incorrect  technical  detail. 


necessary/. 

b.  Pert -Cost  or  anv  scheduling  and 
control  method  should  be  used  only 
as  an  aid  to  management.  Far  too 
often,  management  uses  formal 
techniques  as  a substitute  for 
"real"  management.  .At  best, 
techniques  such  as  PERT  tell  only 
part  of  the  stor*'-  - usually  the 
better  part.  However,  if  used 
properly,  Pert-Cost,  or  its 
equivalent,  is  a necessary  manage- 
ment tool  for  controlling  large 
sofrware  developments. 

c.  Management  cannot  dictate  •-i.-inatural 
schedules  and  e.''oect  a working, 
maintainable  program  "o  result. 

Poor  programs  are  often  a uirect 
result  of  either  inexoer ienced 
personnel  or  forced  schedules. 

d.  Because  the  early  phases  of  software 
development  have  no  visable  output 
except  documentation,  major  mile- 
stones must  be  associated  with  a 
completed  documentation  package. 

A defined  milestone  with  no  visable 
output  is  a useless  milestone  for 
menagement  control.  In  order  to 
Insure  proper  reviews,  documentation 
required  fbr  each  major  milestone 
must  be  defined  in  detail  before 
development  starts. 

e.  Management  should  expect  that  5".  of 
all  sofrware  development  will  not 
work  at  all  and  must  be  redesigned. 
It  will  be  poorlv  documer.tad  and 
employ  unacceptable  design  tech- 
niques. This  fact  will  be  true 
regardless  of  published  standards 


g.  Once  a program  is  working  in  che 
field,  changes  should  never  be  made 
unless  absolutely  necessary. 
Arbitrary  enhancements  or  modifi- 
cations induce  software  bugs  and 
thus  should  be  .svoided. 

h.  Software  costs  can  be  mini-mized  by 
finding  design  bugs  early  in 
development.  Using  IBM's  concept 
of  'Structured  Valk  Through",  most 
design  bugs  can  be  found  prior  to 
testing.  The  cost  oi  software  bugs 
increases  significantly  after  a 
program  responsibility  passes  from 
the  initial  designer  to  a design 
maintenance  gro.up . 

A technical  review  of  det.iilcd  code 
is  required  prior  to  machine  testing 
This  review  can  be  performed  bv  the 
programmers  i.mmediace  suoervisor,  a 
chief  programmer  (if  teams  are 
employed)  or  by  a small  group  of 
peer  programmers . 

1.  Maivagemenc  cannot  assume  that 
programmers  '<now  how  to  design 
software  properly.  Many  exnerienced 
programmers  (and  most  inexperienced 
prograinntrs)  tend  to  generate 
tricky,  complex,  tight,  hard  to 
understand,  and  poorly  documented 
code . 

3ecalled  design  and  coding  standards 
must  be  established  and  followed  in 
order  for  the  resultant  program  to 
be  maintainable  at  a reasonable 
cost . 

j . Oesign  Maintenance  programmers 

require  a higher  level  of  experience 
Chen  that  required  for  original 
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■ ‘ design.  Hiscorlcally  managenen; 

cends  co  place  less  experienced 
progranmers  in  design  naincenance. 
This  is  a coscly  and'  dangerous 
aiscake. 

A Oeveloonen:  Process 

Using  these  and  ocher  ecpirically 
derived  guidelines  a software  developcenc 
process  has  been  derived.  This  process.  . 
which  covers  all  phases  of  development  from 
initial  planning  to  design  maintenance  is 
summarized  in  Chart  3 . 


development  dollars’ include  all  expenditures 
for  planning,  specification,  design,  code 
and  test. 

Since  the  completion  of  coding  is  a 
major  milestone,  it  requires  both,  a tech- 
nical and  managerial  review.  As  shown  in 
Chart  3.  the  technical  review  comes  first. 
This  review  is  organized  co  detect  most  "bad'' 
code.  The  resultant  code  should  be  simple, 
straight  forward  and  easy  co  understand. 

Vhere  applicable,  structured  coding  tech- 
niques should  a'.so  be  employed. 


The  process  recognizes  the  need  for  rwo 
types  of  design  reviews:  technical  and 
aanagerial.  Managerial  reviews  are  held  at 
major  development  milestones.  The  object  of 
these  reviews  are  three-fold:  review 
commercial  documentation  which  is  scheduled 
for  completion  at  the  specific  milestone; 
review  development  expenditures  and  schedules 
- both  historical  and  forecasted;  approve 
continued  development.  Tec’nnical  reviews  are 
much  more  detailed  chan  management  reviews 
and  do  not  consider  schedules  or  budgets. 

The  primary  responsibility  of  a technical 
review  is  to  analyze  the  same  commercial 
documentation  presented  co  management  but  in 
a very  detailed  manner.  Technical  reviews 
are  always  held  prior  to  management  reviews 
and  must  be  successfully  completed  before 
Che  management  review  convenes.  To  illus- 
trate Che  difference  between  these  two 
reviews  we  will  use  the  completion  of 
"coding''  as  an  example  of  meeting  a major 
milestone. 

Sefore  looking  at  the  contents  of  .a 
code  review  it  is  important  co  understand, 
how  a program  is  constructed.  Programs  are 
divided  into  successively  smaller  sections 
called  subprograms,  modules,  segments.  This 
division  into  simpler  parts  allows  a complex 
program  co  be  defined,  developed  and 
maintained.  Vhereas  a large  program  may 
require  30  designers,  a large  subprogram  may 
require  5 designers.  A module  is  usually 
assigned  to  one  designer.  As  a further 
simplification,  the  designer  divides  his 
module  into  individual  fxinctions  called 
segments.  Segments  are  then  coded. 

In  suasnary,  programs  are  built  using 
subprograms . A subprogram  performs  a major 
system  activity.  .An  average  subprogram 
consists  of  about  3.000  source  lines.  Sub- 
programs are  built  using  modules . A module 
performs  a sec  of  very  closely  related 
functions  and  consists  of  about  300  source 
lines.  Modules  are  built  using  scynencs . 

A segment  pertoras  one  function  ana  consists 
of  30  source  lines.  Segments  arc  built  from 
source  lines  (or  instructions!)  . Source 
Lines  are  the  most  basic  building  block  in 
software  design. 

Coding  Review 


Experience  indicates  chat  this  review 
should  cake  place  at  about  the  point  where 
65”  of  all  software  development  dollars  have 
been  expended.  In  this  case,  total  software 
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Unit  tests  should  be  made  available  at 
Che  code  review  and  subset  of  these  tests 
"mentally"  executed  by  the  reviewing  body. 
This  process,  when  combined  with  code  reading 
(a. process  where  code  logic*  and  code  format  ' 
is  scrutinized  by  a programmer  ocher  than 
original  designer) , can  detect  up.  co  90”  of 
coding  and  s:mtax  errors . Code  reading  and 
mental  testing  should  always  be  employed 
prior  CO  object-machine  testing  in  order  co 
minimize  usage  of  expensive  machine  time. 
Development  cost  recuired  co  detect  an  error 
by  reading  code  is  approximately  equal  to 
257. -chat  required  co  detect  an  error  via 
machine  debugging. 

Probably  the  most  important  concept 
associated  with  this  technical  review  is  tne 
actual  commercial  coda  is  reviewed.  The 
flow  and  structure  of  the  code  is  analyzed 
for  design  flaws  and  mis-incerprecacions . 

The  review  body  is  'Kept  smell  - always 
Including  the  chief  programmer.  In  some 
reviews  additional  people  may  attend:  coders 
who  are  responsible  for  software  directly 
Interfacing  with  the  module  being  reviewed, 
first  level  supervisor  and  Che  programmer 
who  will  be  responsible  for  maintaining  the 
software  module  after  development  is 
complete . 

Once  the  technical  review  is  complete, 
management  should  be  given  a presentation 
summarizing  the  results  of  the  preceding 
reviews;  the  technical  review  held  after 
completion  of  design  and  the  technical  review 
held  after  completion  of  code. 
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The  r.ar.ai?eta«nc  review  is  held  afrer 
cosplecion  of  coding  racher  rhan  afcer 
conplecion  of  design  so  phac  T.ore  accurace 
inscrucpion  espiT.aces  will  be  available  (see 
Chart  5).  Since  these  reviews  are  held  or. 
a subprogram  basis,  the  first  review  will  be 
held  the  early  part  o:  the  "design  and  code 
complete"  curve  shown  in  Chart  6.  It  is 
good  practice  to  require  test  plans  and  all 
commercial  documentation  to  be  brought  to  the 
management  review  - for  psychological 
purposes  only.  Management  should  be  con- 
cerned resolving  non- technical  problems  and 
with  correlating  results  of  this  review  with 
other  project  reviews  (a  large  software 
project  will  have  coding  reviews  spread  out 
over  a year  Or  .uore)  . 

The  major  objective  of  a management 
review  is  to  maintain  schedules  and  budget 
by:  shifting  manpower  from  less  important 

activities  to  critical  tasks,  cancelling  or 
delaying  features,  allowing  standard 
practices  to  be  short-cut  and  if  all  fails 
CO  iccediacely  publish  a schedule  slip  or 
budget  increase. 

A second* obj ective  of  the  management 
review  is  to  insure  that  the  project  is  still 
profitable.  It  should  be  noted,  ho'wever, 
that  65".  of  total  development  dollars  have 
been  spent  by  the  ti.me  the  coding  review 
occurs.  Ac  this  staae  of  development,  custom- 
er cormiitments  have  orobably  been  solidified 
- to  suc.h  an  extant  that  even  unproficaole 
projects  continue  to  cor.tletion. 

The  management  review  process  ’which 
occurs  after  code  is  complete  is  probably 
the  most  comprehensive  as  far  as  technical 
concent.  Earlier  management  reviews 
concentrate  more  on  "should  the  project 
continue”.  For  this  reason  management 
reviews  are  concentrated  around  the  specifi- 
cation stage. 

ThE  3E'/E:.0PM£NT  PROCESS 

This  section  will  describe  a C'.’pical 
flow  through  the  development  cvcle.  The 
description  is  somewhat  detailed  out  this 
emphasis  is  justified  since  the  development 
process  is  the  heart  of  soffware  management. 
If  a "good"  de*/eiopment  process  is  combined 
with  a "good"  scheduling  and  cost  control 
cools,  management  will  experience  a high 
percentage  of  successful  softvar  projects. 


dollars.  It  is  at  this  fi-sc  revie*w  that 
most  marginal  or  non-prof itable  projecc%  will 
be  "turned  off"  by  management. 

The  types  of  information  brought  to 
management  by  t.he  project  engineers  are: 

- Product  description  - description  of 
system  features  as  seen  by  the 
customer . 

- High  level  description  of  ho’w  major 
features  will  be  implemented. 

- Software  hierarchy  chart  where  each 
element  in  the  chart  represents  about 
3.0C0  instructions  (a  subprogram). 

- Estimates  of  schedule,  development 
cost,  memo'ry  size,  real  time,  market' 
potential,  material  and  system  cost. 

Armed  with  the  above  information, 
management  can  determine  if  the  reviewed 
project  deserves  further  funds.  The 
estimates  vary  in  accuracy  depending  on  the 
completeness  of  the  original  product 
description  ana  the  esti.macers'  experience. 

* 35"  seems  to  be  an  average  accuracy  figure.  • 
Chart  5 indicates  that  management  can 
improve  upon  this  accuracy  by  increasing  the 
program  size  estimation  'which  they  receive 
from  their  technical  staff  by  30". 

If  management  approves  further  devel- 
opment, a project  leader  is  appointed.  Chief, 
programmers  are  also  selected  at  this  time. 
These  people  will  execute  the  next  stage.  The 
project  leader  coordinates  all  project 
activities  - both  hardware  and  sofrware.  He 
is  responsible  for  integrating  schedules  of 
all  involved  development  groups.  The  project 
leader  also  controls  all  system  level 
documents  - such  as  the  specification  of  how 
the  system  (or  program)  is  to  operate  sr.d 
what  features  are  included.  The  chief 
programmer  Is  a technical  leader  who  will 
eventually  be  in  charge  of  a team  of 
programmers.  Large  projects  will  have  one 
project  leader  and  many  chief  programmers 
(a  chief  programmer  team  should  be  limited 
to  five  progratmrers) . In  small  projects, 
requiring  only  one  chief  prograimer.  the 
project  leader  and  chief  progracser  can  be 
the  same  person. 

Phase  2 - Soecifiearicn 


Phase  1 - Planning 

The  first  phase  of  development  is  called 
the  planning  stage.  During  this  stage  the 
customer  generstes  his  requirements.  Normally 
the  custom.er  will  work  directly  with  project 
engineers.  If  a project  requires  only 
sofrware  effort,  this  engineer  should  be  an 
experienced  systems  programmer.  If  a project 
requires  hard’ware  and  sofrware.  the  project 
engineer  must  be  knowledgeable  in  both  areas. 

A primary  objective  of  planning  is  to 
quickly  generate  a "hiih  level  package"  for 
management  revie*w.  This  phase  should  not 
spend  more  than  I",  of  total  development 


The  next  management  review  will  occur 
afcer  10^.  of  the  total  development  dollars 
estimated  during  Phase  1 have  been  expended. 
The  output  of  this  phase  is  generated  by  the 
project  leader  and  chief  programmer (s > . All 
technical  documentation  is  generated  in  a 
format  acceptable  for  commercial  release. 

Since  the  project  is  still  in  early 
phases  of  development,  the  probability  of 
managerial  rejection  is  high.  During  this 
phase,  firm  marketing  escl.-nates  arc  estab- 
lished. Cost  and  schedule  accuracies  arc 
improved  from  35”.  to  15",.  The  orimarv 
.''t ; cc:  o:  this  ise  is  ;c  'or ir j a; curate 
information  to  mar.agament  fer  project 
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approval  of  disapproval.  Tha  probabllicy 
chac  a projecc  will  ba  approvad  ac  rhis  sraga 
and  cumed  off  ac  a later  3caj;e  should  be 
small,  for  afcar  cha  lOZ  phase  devalopmanc 
coses  Incraasa  vary  rapidly  and  a cocal  man- 
power comicmanc  is  made. 

The  ouepue  aanaracad  during  the 
specif icacion  stage  consists  of  tha  following 
cachnical  documants : 

a.  Subprogram  hierarchy  chart  showing 
control  flow  between  modules. 

b.  Tima  sequence  description  explaining 
cha  sequence  in  which  modules 
execute  in  order  to  perform  major 
system  features. 

c.  Subprogram  prologues  which  describe 
cha  functions  performed  by  each 
subprogram,  the  inputs  to  each  sub- 
program and  cha  outputs  from  each 
subprogram. 

d.  Module  prologues  which  describe  the 
functions  performed  by  each  module, 
module  inputs  and  module  outputs . 

e.  A general  layout  and  description  of 
all  data  used  by  the  program. 

f.  A description  of  all  man/machine 
interfaces . 


The  administrative  documents  generated 
during  the  specification  stage  commit  to 
specific  values  for;  memory  size,  real  time, 
devclopmenc  cost,  development  schedule,  and 
resource  requirements . These  technical  and 
administrative  documents  form  a "contract" 
by  the  developing  organization. 

Phase  3 - Design 

After  Phase  2.  development  activities 
acceTerate  very  rapidly,  tf  the  technical 
base  established  in  Phase  2 is  generated 
properly,  a significant  number  of  programmers 
can  be  placed  on  the  sofrware  problem  early 
in  Phase  3 . 

The  Phase  3 review  occurs  after  507.  of 
total  development  costs  specified  in  Phase  2 
have  been  expended.  Phase  3 is  a turning 
point  in  that  most  managers  no  longer  have 
the  ability  (or  time)  to  perform  a detailed 
review  of  work  performed  and  thus  must  rely 
on  technical  "walk-throughs"  (IBM  termin- 
ology) . The  output  of  Phase  3 contains  m 
code.  This  Is  an  Important  concept.  The 
walk-through  is  a formal  review  technique 
where  test  inputs  are  generated  and  "eye- 
balled"  through  the, design  to  verify  its 
accuracy.  Experience  shows  chac  coding  is 
started  far  coo  early  in  most  software 
projects  and  because  of  this  a large  percent- 
age of  code  is  redone  before  the  software 
passes  evaluation. 

The  technical  "walk- through"  performed 
ac  Che  conclusion  of  Phase  3 is  attended  by 
Che  project  leader,  (he  is  responsible  for 


program  specification),  chief  programmer, 
interfacing  prograrmers  and  the  supervisor. 
The  output  of  Phase  3 consists  of; 

a.  An  update  of  all  technical 
documentation  - described  in  Phase 
2. 

b.  Module  hierarchy  charts  which  show 
control  flow  between  segments. 

c.  Prologues  which  describe  the 
function  performed  by  each  segment 
within  a module.  Definition  of  all 
data  inputs  to  each  segment  and 
data  outputs  from  each  segment. 

d.  Test  plans  required  to  test  the 
operation  of  each  module. 

e.  Detailed  data  maps  and  description 
of  each  cable  and  item. 

All  these  documents  with  exceocion  of 
module  test  plans  are  generated  for 
coonercial  release. 

This  review  is  organized  to  detect  all 
"bad"  program  modules.  "Sad"  in  the  sense 
of  poor  structure.  Ve  require  programs  to 
be  built  rrom  code  blocks  - called  segments. 
Each  code  block  should  perform  only  one 
function  and  be  fully  documented;  program 
modules  are  built  from  these  code  blocks 
using  rules  of  structured  design.  "Bad"  in 
the  sense  of  incorrect  or  pooly  defined 
interfaces : modules  are  also  reviewed  to 
insure  correct  inter-module  interfaces  and  to 
assure  chac  functions  performed  are  in 
agreement  with  program  specification . 

Phase  4 - Coding 

This  is  the  coding  phase  chac  was 
previously  explained.  Since  the  entire 
program  is  completely  specified  after  Phase 
3 segment  coding  can  cake  place  in  parallel 
manner.  Top  down  coding  need  not  be 
employed.  Top-down  coding  will  be  used  only 
in  those  developments  where  manpower  avail- 
ability indicates  that  top-down  will  not 
delay  projecc  completion. 

From  an  administrative  standpoint,  all 
estimates  are  refined.  An  accuracy  figure 
of  better  chan  10^  should  be  available  for 
memory  size,  'real  time  requirements  and 
development  costs.  An  accuracy  of  better 
chan  57.  should  be  available  for  schedules. 

Pha.se  5 - Testing 

This  pha.se  con.slHix  of  four  st.'igeM  of 
software  testing. 

First,  each  segment  is  tested  to  Insure 
its  singular  function  works  as  specified  in 
Phase  3.  Second,  the  segments  are  strung 
together  to  insure-  that  groupings  of  segments 
(called  modules)  work  as  specified  in  Phase 
3.  Both  of  these  testing  activities  are 
performed  by  the  design  prograticner.  Either 
top-down  and  botcom-up  testing  may  be  em- 
ployed. Modules  which  are  tested  earlier 
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are  hiah-level  concrol  segnears . harivare 
Interface  segrtents  cr  segments  prone  to 
interface  errors. 

After  the  chief  orogratmer  agrees  that 
unit  tasting  and  string  testing  is  conolece. 
the  chief  orogra-ttser  together  with  the 
tiaintenanca  progratrr.er  (the  person  who  will 
be  responsible  for  rr.aintanance)  verify  the 
operation  of  each  subprogram.  The' mainte- 
nance progratir.er  being  an  experienced 
programr.er  is  responsible,  along  with  the 
chief  prograttr.er , for  generating  (or  at  least 
agreeing  with)  the  integration  test  plan 
which  checks  the  subprogram  operation.  The 
generation  of  integration  tests  should  start 
as  soon  as  possible  after  Phase  2.  lariv 
development  of  these  test  plans  helps  uncover 
"holes"  in  the  sofrware  specification. 

Since  the  maintenance  progrataner  and  the 
chief  prograszser  should  report  to  different 
first  line  super'-'isor , an  important  cross- 
check is  established.  At  completion  of 
integration  testing,  the  maintenance  orograo- 
mer  must  agree  that  the  subprogram  works 
according  to  the  specification  ger.eracad  in 
Phases  2 and  3.  The  maintenance  progrataer 
also  formally  agrees  that  the  subprogram 
meets  comciercial  standards'  well  documented, 
structured  design,  easy  to  understand  code. 
Because  the  maintenance  orogranmer  will  be 
respon.sible  for  the  suiiortigr.cm  .if Lor  chi> 
original  designer  rel  inoui.shes  respon.sibt  II  Cv, 
he  will  not  accept  the  subprogram  wichout 
serious  analysis.  Unacceptable  subprograms 
are  icraediately  brought  to  the  attention  of 
second  level  managers  and  the  project  leader. 

The  last  stage  of  design  testing  checks 
the  inter-operation  of  all  subprograms'.  A 
team  approach  is  used  for  this  "total 
program"  or  system  test.  The  team  consists 
of  selected  chief  progratisers , senior  mainte- 
nance programmers,  and  evaluation  engineers, 
•All  members  are  involved  in  generating  and 
executing  the  test  plan.  The  evaluation 
engineer  is  a svstem  expert.  In  chose  areas 
where  hardware  and  software  are  involved,  the 
evaluation  engineer  should  be  knowledgeable, 
from  a svstem  star.aooinc,  of  .ill  area*. 

During  system  testing,  -ill  functlon.s  if  the 
program  are  tasted  under  stress  conditions. 
Errors  are  placed  into  operational  software, 
data  structures  and  svste.m  hardware.  External 
drivers  are  employed  to  simulate  heavy  "user" 
activity.  The  objective  of  system  testing 
is  to  make  the  system  fail. 

Ac  cha  conclusion  of  system  testing,  the 
evaluation  engineer  must  formally  agree  thee 
Che  program  successfully  passed  system 
testing.  This  acceptance  is  accompanied  by  e 
list  of  outstanding  problems.  Acceptance  is 
contingent  upon  these  problems  being  resolved. 
The  program  list  is  passed  on  to  second  level 
management  end  the  project  leader  to  insure 
prompt  resolution.  A formal  follow-up 
procedure  must  be  established  to  give  these 
problems  appropriate  prloritv.  During  system 
testing,  and  far  the  life  of  the  program  - 
all  software  bugs  are  formally  documented  and 
formall'.'  scheduled  for  resolution.  Surrtary 
data  i:-'dicatlng  "bugs  per  nodule"  is  fad  back 


to  design  super'^isors . This  data  allows  ‘ 
supervisors  to  evaluate  and  improve  design 
procedures  as  well  as  progratsming  staff. 

After  system  testing  is  comolete.  the 
responsibilitv  for  moJifving  the  source 
program  is  relinquished  by  the  design  program-  i 
met  (and  chief  orogrsmmer ^ ana  passes  or.  to 
the  maintenance  program.m.er . After  svstem 
testing,  tne  program  should  work  In  its 
intended  environm.ent . The  software  mainte- 
nance phase  new  begins  and  the  total  system 
is  evaluated  by  an  independent  group  of 
system  experts.  The  primary  function  of 
"evaluation"  is  to  insure  that  the  svstem 
meets  all  customer  requirements  from  a 
feature  standpoint,  a man, machine  interface 
otandpoint,  'a  docum.encacior.  standpoint,  and 
real-time  standpoint.  During  evaluation,  a 
complete  sec  of  acceptance  tests  are  executed. 
Design  problems  are  resolved  by  c.he  mainte- 
nance programmers.  The  effort  expended  in 
evaluation  testing  is  not  considered  part  of 
the  software  develooment  oosc. 
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The  successful  development  of  a large 
software  controlled  svstem  is  at  best  only  a 
partially  completed  task.  Although  many  feel 
that  the  mar.agem.er.c  of  large  software 
development  is  mvseerious.  or  at  least  little 
iincterscoo(.l ; rne  long  term  control  inii 
m/I  Int  cniincc  of  l.iri'.c  prcii’.riims  t.s  ovi-n  :noro 
TVS  ter ious  . 


The  management  process  which  is  used  to 
maintain  a program  after  completion  of  design 
is  called  "software  conf igyracior.  management". 


This  process  allows  management  to 
control  the  maintenance  and  manufacture  a 
software  package  (or  program) . The  process 
also  allows  management  to  control  modifi- 
cation to  this  package  and  to  insure  software 
changes  are  m.ade  coincident  with  associated 
hardware  changes.  Programs  should  be  placed 
under  configur  ir  Ion  m.in.igement  af'or  •i;ih- 
progr  ini  i nCt-i’.r  i ? ion  'ml  befun-  cunri!,"  rm  .'f 
system  rs  (.  i ni’ . After  inn  f 1 ivir  it  i m:  ii'ii'r>l 
takes  efiect.  .ill  code  ind  iloeumcoti:  icn 
changes -must  be  accomnariod  by  a sunervisorv 
approved  form  I.ndicacing  reason  for  change 
and  a test  plan,  if  applicable. 


The  controlled  software  package  is  sold 
CO  Che  customer.  It  is  celled  e "program 
version".  If  different  sections  of  the 
market  require  slightly  different  programs 
- each  section  is  supplied  e different 
program  version.  One' of  the  large  real-time 
programs  described  earlier  exists  as  four 
different  versions.  Two  of  the  versions 
ser-ze  the  same  market  but  contain  different 
feature  packages.  The  ocher  rwo  versions 
serve  different  markets  (outside  I'SA) . Each 
of  the  four  versions  contain  approximately 
1,D00  modules.  BO",  of  the  modules  are  corw.cn 
among  the  four  versions. 


Given  this  situation,  configuration 
managemen:  allows  us  to. 
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■ a.  Maintain  tha  comson  aiodulas  only 
onca.  Thus  1,600  modules  naed  to 
ba  maincainad  rachar  than  4,000 
nodules.  A much  larger  design 
maintenance  effort  would  be  required 
if  coinnon  modules  ware  not  recog* 
nizad. 

b.  Design  new  versions  using  existing 
common  modules . 

c.  Insure  that  corrections  made  to 
modules  in  one  version  are  reflected 
in  all  versions . 

d.  Allow  systems  Co  recrofH  from  one 
version  co  another  version  wichout 
affecting  customer  service. 

a.  Insure  that  all  software  changes 
have  proper  management  approval 
and  are  thoroughly  tested  before 
being  sent  to  the  field. 

Under  configuration  management  a version 
is  updated  either  annually  or  semi*annually 
by  issuing  reassembled  program  loads.  When 
this  is  done, 'each  site  employing  the 
specific  version  must  place  the  updated  load 
in  its  machine.  This  update  is  called  a 
..  release.  If  important  changes  must  be  made 
to  the  commercial  program  at  a faster  race 
than  releases  are  generated,  than  patches 
may  be  sent  to  the  field  in  the  form  of  point 
releases.  Experience  indicates  that  point 
'*  releases  occur  at  six  week  intervals.  Charts 
9 and  10  show  releases  of  the  four  versions 
T of  the  above  mentioned  program. 

- ■ ' A release  contains  both  design 

corrections  and  new  features.  Versions 

• remain  active  (i.e.  are  updated  with  re- 
leases) for  approximately  four  years.  During 
this  period  releases  co  these  versions  occur 
at  approximately  the  following  intervals: 

^ - Three  month  interval  for  first  six 

• months. 

- Six  month  interval  for  next  eighteen 
*'  months. 

’*  - One  year  interval  for  next  two  years. 

• Controlling,  approving  and  monitoring 
I software  fixes  to  a commercially  released 

• • program  is  an  important  aspect  of  design 

maintenance.  This  function  is  most 
r efficiently  performed  by  a review  group 

consisting  of  members  who  are  close  co  both 
technical  detail  and  management  philosophies. 
Design  maintenance  supervisors  are  excellent 
candidates  for  this  group.  This  group  both 
schedules  manpower  to  resolve  software  bugs 
and  insures  chat  reported  problems  arc  not 
enhancements  co  the  original  software 
specification. 

The  above  controls  are  strictly  enforced 
for  non-cricical  problems.  These  controls 
must  be  relaxed  when  serious  service  affec- 
ting software  problems  arc  suspected  to  be 
in  a released  program.  To  resolve  these 
, problems  in  large  sofeware/hardware  systems 
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a specially  trained  team  (called  an  ATTACK 
TEAM)  must  be  established.  This  group  of 
specialists  is  given  specific,  system  level, 
training  aimed  at  quickly  resolving  complex 
problems  which  exist  in  an  on-line  site'. 
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Program  Modifications  After  Commercial 

Release 

As  was  mentioned  earlier,  more  develop- 
ment effort  is  spent  in  the  10  years 
following  a development  than  during  the 
initial  development.  This  characteristic 
docs  not  seem  to  be  particular  co  a specific 
application  or  co  a specific  design  approach. 
Programs  require  extensive  maintenance  after 
the  first  connercial  release  for  four 
reasons ; 

a.  Changes  in  Man/Machine  Requirements 
- After  a program  reaches  the  field, 
customers  find  more  effective  ways 
co  interface  with  the  program  - 
either  for  maintenance  or  opera- 
tional reasons.  Implementing  these 
changes  is  considered  part  of 
design  maintenance. 
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b.  Snail  Feature  Addirions  ar.d  Er.hance- 
aencs  - As  a rrogran  natures  In  the 
field,  potential  customers  retuire 
added  snail  features  or  enhancer.eats 
to  aeet  their  specific  enviroanent. 
Since  this  acti'/ity  is  necessary  for 
continued  sales  it  is  considered 
part  of  design  naintenance . 

c.  Latent  Software  3ugs  - No  natter 
how  carefully  a program  is  tested 
prior  to  the  first  cotrr.ercicl 
release,  a significant  number  of 
software  bugs  will  be  found  in  large 
real-tine  programs  as  the  program 
faces  different  user  environments. 

d.  Induced  3ugs  - New  software  which 
enters  the  field  always  contains 
new  bugs.  .Analyzing  the  source  of 
software  bugs  found  in  a connercial 
program  indicates  that  50".  have 
always  been  there,  50".  are  due  to 
new  features,  03”.  are  generated  by 
fixing  other  software  prob.lems,  19". 
are  miscellaneous. 

Chart  11  shews  the  percentage  of  program 
modules  which  were  modified  for  one  or  the 
above  four  reasons  during  the  first  two  years 
after  commercial  introduction.  Program  3 
has  seen  45”.  of  its  modules  change  during  the 
first  two  years.  This  particular  svstem  has 
had  few  feature  en.hancenents  aeded  to  its 
base  during  its  maintenance  period.  .As  can 
be  seen  in  Chart  11.  module  changes  to 
program  3 are  tending  to  level. 

CHART  n 

P€BCENT*GE  Of  MOOUt-ES  CHANGED  CXS  ’0 


Program  A has  seen  over  707.  of  its 
modules  change  during  this  same  two  year 
period.  This  program  has  encountered 
significant  feature  enhancements  and  has 
also  enjoyed  a large  market  - which  has 
forced  the  addition  of  many  small  enhance- 
ments to  meet  customer  requirements  and  has 
forced  the  program  to  face  many  different 
external  environments. 


changes  before  thev  are  released  to  the 
field. 

The  configuration  management  controls 
call  for  four  levels  of  testing  of  all 
program  m.odifications  before  thev  are 
released  to  the  field  in  the  form  of  i 
version.  As  shown  in  Chart  ID,  those  tests 
ihe lude : 

- Original  designer  test  (not  in  chart)  . 

- Evaluation  test  on  laboratorv  model 
for  five  weeks. 

- Field  Cost  on  commerciaL  off-line 
sire  fur  ; wo  wcck.s  . 

- Field  test  on  commercial  on-Ii.ne  sice 
for  two  weeks . 

Chart  12  shows  statistics  indicating 
the  number  of  program  patches  that  were  made 
to  each  of  two  different  systems.  These 
changes  were  sent  to  the  field  in  the  form, 
of  point  releases.  .As  can  be  seen  i.n  this 
chart,  a formal  test  plan  is  required  per 
software  change.  The  chart  also  s.nows  that 
all  four  levels  of  testing  are  required  in 
order  to  achieve  a program  acceptable  for 
field  distribution.  In  Chart  12.  System  3 
did  not  test  patches  at  an  off-line 
sice.  Because  of  this,  eight  coding 
problems  were  found  at  the  on-line  test  site. 
System  .A  did  have  an  off-line  test  site 
where  most  of  the  software  bugs  were  found 
and  corrected.  Since  these  statistics  have 
been  compiled  a fifth  level  of  cross-check 
has  been  added  to  the  above  process.  This 
cross-check  occurs  after  original  designer  ■ 
testing.  A separate  grouo  now  visuallv 
inspects  and  approves  all  source,  updates  and 
all  test  plans  orior  to  initial  evaluation 
testing.  This  group  has  been  able  to  reduce 
the-  number  of  errors  reaching  evaluation 
testing  by  -.0". 
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Software 

Sugs  Found  During 

Design  Mainte- 

Since  many  modules  arc  being  modified 
during  the  earlv  period  of  comercial 
availability,  conf izuration  m.anagem.snt  must 
enforce  both  tight  control  of  sofrware 
changes  as  well  as  extensive  testing  of  these 


Early  phases  of  sofrware  debugging  are 
aiirad  at  finding  simple  bugs.  Later  stages 
of  testing  usually  concentrate  on  the  m.o're 
complex  problem.^.  Code  reading  is  an  a:<amsle 
of  testing  which  is  executed  very  early  in 
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Che  debugging  phas«  • even  prior  co  InlElal 
machina  ccsclng.  7ht  mora  complex  software 
problems,  however,  usually  show  uo  during 
system  testing  or  after  the  program  has  been 
placed  into  service.  The  development  cost 
required  to  detect  and  resolve  a software 
bug  after  it  has  been  placed  into  service  is 
thirty  times  larger  than  the  cost  required 
to  detect  and  resolve  a bug  during  the  early 
"code  reading"  phase. 

This  large  cost  difference  cannot  be 
explained  solely  by  the  fact  that  the  prob- 
lems involved  are  of  different  conolexitv. 
for  if  we  compare  the  development  cost 
required  to  solve  problems  of  equal  complex- 
ity - the  difference  is  still  fifteen  to  one. 
This  concept  points  to  the  general  management 
guideline  that  software  bugs  should  be  found 
at  the  earliest  possible  stage  of  testing  in 
order  co  minimize  overall  development  costs 
and  development  schedules ; it  is  more 
efficient  to  find  and  correct  a simple  coding 
mistake  during  early  code  reading  chan  during 
machine  test  (by  a factor  of  four  to  one)  or 
during  design  maintenance  (by  a factor  of 
fifteen  co  one) . 

• • 

Software  bugs  cost  more  tu  corri-ct  iifcer 
a program  has  been  released  to  i.!ic  customer 
chan  during  early  phases  of  testing  for  the 
'*  following  reasons; 

a.  After  commercial  release  - problems 
are  usually  more  complex. 

b.  After  coqtaercial  release  - problems 
are  reported  as  system  malfunccions ; 
an  effort  must  be  spent  to 
translate  probl«n  into  a software 
bug. 

c.  After  commercial  release  - many 
problems  are  resolved  by  design 
maintenance  programmers  rather  chan 
the  original  designer.  Design 
maintenance  programmers  must  spend 
effort  reviewing  detailed  code. 

d.  After  comnercial  release  - problems 
require  more  definition  and  more 
formal  documentation.  Formal  test 
plans  and  multi-level  testing  must 
be  performed  co  insure  chat  accurate 

, corrections  reach  the  field. 

c.  After  commercial  release  - problem 
resolution  must  share  the  heavy 
overhead  cost  for  configuration 
management. 

ECONOMIC  ANALYSIS  OF  SOFTWARE  DESIGN 

It  is  imperative  chat  all  software 
development  be  aimed  at  one  goal  - co 
maximize  profits  for  the  supporting  company. 
Often  this  objective  is  lost  among  a complex 
entanglement  of  technical  goals.  To  achieve 
Che  objective  of  profit  maximization,  four 
factors  must  be  considered:  Oevelopmcnc  and 
maintenance  expenses,  sales  dollars,  system 
cost,  availabllicv.  Since  costs  are 
constantly  changing  due  co  inflation  and 
technological  advances  in  hardware 


(especially  solid  state  memories)  projected 
costs  rather  chan  current  costs  should  be 
used  when  calculating  all  expenditures  and 
income . 

Most  software  developments  are  both 
initiated  and  completed  without  managerially 
established  objectives.  Some  of  the  better 
m.inageu  sofcv.ire  projects  are  developed 
with  such  r.an.a^'emenc  objectives  .is 
maximize  real  time  or  minimize  storage. 
Although  these  cechnicallv  oriented  objec- 
tives are  better  t.han  none  at  .all  they 
usually  fall  short  of  meeting  the  universal 
objectives  of  all  managers  - optimization  of 
profits. 

A large  void  currently  separates  soft- 
ware management  and  the  concept  of  profit 
optimization.  In  most  cases  the  reason  for  ■ 
this  gap  exists  because  too  few  software 
leaders  are  trained  managers  and.  inversely, 
coo  few  good  managers  understand  the 
specifics  of  software  development. 

Unfortunately,  there  is  not  one 
universal  wav  co  develop  software.  Contrary 
to  the  belief  of  mnnv  m.m.u',orM , all  software 
need  nor.  be  modiil.ir,  sinieturod  or  pcnerle. 

Wo  e.'in  .nil  think  of  mnnv  ex.nmnle’<.  One 
obvious  example  l.s  a program  which  he 

designed  by  inexperienced  programmeta  and 
used  onlv  one  time  - for  a special  on-line 
retrofit  process.  A good  manager  must  not 
only  be  aware  of  all  technologies  associated 
with  software  development,  but  he  must  also 
take  evet^r  opportunity  to  "test  the  water" 
by  experimenting  with  development  concepts. 
Thus,  with  experience,  a good  manager  will 
be  able  to  match  various  concepts  of 
software  development  with  soecific 
orojecc  requirements  and  available 
resources . 

Certainly,  in  an  idealized  world, 
software  should  be  developed  using  high-level 
language.  Software  architecture  should  be 
modular  and  code  should  be  structured. 
Docuinencacion  should  be  simple  yet  effective. 
Chief  programmer  concepts  and  egoless  • 
programming  should  be  practiced.  Extensive 
integration  and  evaluation  test  plans  are  a 
must . 

In  Che  real  wprld,  however,  these 
idealized  concepts  must  be  balanced  against 
the  fact  chat;  compilers  may  not  exist, 
programs  may  only  be  executed  once  then 
disregarded,  talent  mix  may  not  justify  a 
chief  programmer.  Time  and  budget  may' not 
allow  extensive  evaluation. 

Thus,  software  management  is  like  all 
other  forms  of  menegement;  it  can  be 
effectively  practiced  only  by  chose  individ- 
uals who  have  become  both,  mature  techno- 
logists as  well  as  mature  managers. 

In  order  co  limit  the  scope  of  this 
paper  I will  consider  the  "profit 
opcimlzacion"  concept  for  a specific  t:vpe  of 
software:  a program  which  executes  in  real- 
time. contrtllLr.g  the  tperatisns  of  a large 
commercial  hardware  machine. 
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For  chis  prcgrar.,  profit  opcinisatior. 
car.  be  tet  bv  first  assuning  a constant  sales 
price  and  than  by  establishing  ''dollar  '/alue" 
trade-offs  bef-een  the  following  controllable 
factors : 

- Metaory  size  (affects  system  cost)  . 

- Real  tine  (affects  sales  dollars) . 

- Development  schedule  (affects  sales 
dollars)  . 

- Development  hours  (affects  expense)  . 

- Design  maintenance  (affects  expense) . 

- Machine  aaintenance  hours  (affects 
sales  dollars) . 

A dollar  value  can  be  assigned  to  each 
factor.  Management's  task  is  to  minimize 
cash  out-flow  (expense)  and  to  maxi.mice  cash 
in-flow  (sales  dollars  lass  cost  or  sales) . 

As  a simple  illustrative  example'  of  the 
above  optimization  approach  let  us  answer  the 
question:  Should  high-level  language  be  used 

to  develop  the  control  orograr.  tor  a 
switcning  svste.m? 

Typically  this  cuestion  is  answered  by 
a manager,  either  affirmatively  - because  he 
feels  that  high-level  language  is  good,  or 
negatively  - because  he  feels  high-level 
language  is  bad. 

To  use  the  profit  optimization  process 
a manager  will  gather  the  following 
information; 

a.  Memory  Size  - High-level  language 
will  increase  storage  estimates  from 
100,000  words  to  120,000  words.  Cost 
of  memory  is  25c  for  each  word. 
Therefore,  cost  of  using  high-level 
language  is  S3, 000  per  system  sold. 
The  market  life  is  one  year  and  100 
systems  will  be  sola.  The  total 
added  cas'n  out-flaw  for  additional 
memory  is  5500,000. 

b.  Real  Time  - High-level  language  will 
reduce  available  time  by  lOT,.  This 
reduction  can  be  translated  into  a 
reduction  in  sales.  As  a first 
approximation  a linear  reduction  in 
sales  volume  can  be  used  (10% 
reduction) . This  reduction  in  sales 
represents  a reduction  in  net  cash 
in-flow  (sales  dollars  less  cost  of 
sales^of  SIOO.OOO. 

c.  Development  Schedule  - Use  of  high- 
level  language  will  reduce  develop- 
ment interv-ai  by  months  . This 
means  that  the  product  will  be 
available  6 months  earlier.  This 
data  should  now  be  translated  into 
additional  sales  and  consequently 
additional  cash  in-flow.  .As  a first 
approximation  assume  that  active 
sales  life  will  increase  by  6 
months.  With  constant, annual  sales 


c3.-<h  In-flowwiH  incro.i 
or  S500.C00. 

d.  Development  Hours  - High-level 
language  will  decrease  original 
design  hours  by  13”..  .AS3u.ming 
S20.DC  per  design  hour  and  2 hours 
per  assembly  level  instruction, 
development  cost,  or  cash  out-flow, 
is  reduced  bv  5600 , 200 . 

e.  Do.sign  M.sinten.ip.ce  Hours  - Design 
maintenance  effort  is  reduced  bv  5".. 
This  calculates  to  a reuuction  in 
cash  out-flow  of  3100,000  (time 
value  of  money  must  be  considered) . 

f.  Sice  Maintenance  Hours  - Maintenance 
activity  on  each  site  is  not 
affected.  This  area  becomes 
important  when  considering  whether 
improvements  should  be  made  to  man/ 
machine  interfaces  or  whether 
improvements  should  be  made  to  soft- 

. ware  diagnostics. 

•A  profit  optimizacton  decision  is  made 
by  adding  "a"  through  "f". 

.Met  advantage  of  high-level  language 

•a-t-b  + c-td  + e + f 

• -500K  -lOCK  500K  600K  * IDOK 

- S6C0.000 

Thus,  high-level  language,  if  available, 
should  be  employed  in  this  application.  If, 
however,  a compiler  must  be  designed  - its 
development  cost  .must  be  less  than  S6C0.D00._ 

ORGANIZATIOM  STRTCTVRS 

The  structure  of  an  organization  can  be 
analyzed  from  many- viewpoints  What  may  look 
like  a functional  organization  at  a super- 
visory level  in  the  organization  scrccture 
(see  Chart  13)  may  well  Ioo'k  like  a project 
organization  from  a unit-head  level  in  the 
organization.  Thus,  in  order  to  discuss 
organization  structure  one  must  specify  which 
level  in  the  organization  is  being  used  as  a 
reference  point. 

I would  like  to  start  this  discussion 
using  a "department"  or  "director"  level  as 
reference.  In  this  discussion  a department 
i.s  defined  as  a group  whose  sice  v.ir-.es 
between  75  and  250  poopLo.  Ideally  a 
department  should  contain  no  more  th.in  120 
professional  people.  It  is  controlled  by 
three  basic  levels  of  management  which  we 
shall  call  supervisor,  unit  head,  and 
director. 

A department  is  chosen  as  the  primary 
reference  point  since  it  is  the  highest  lavel 
in  a corporats  structure  which  exerts  a 
strong  influence  on  the  day  to  day  work 
activities  of  progratsaers . As  such,  the 
three  levels  of  ceoartment  management  have 
prim'ary  responsibility  for  budgets  and 
schedules  of  activities  being  performed  by 
the  technical  personnel  weehin  che  ceeart- 
ment . 
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CHART  13 

ORGANIZATION  STRUCTURE  OF  A DEPARTMENT 
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Software,  as  with  all  other  foras  of 
development,  should  start  with  a control- 
lable and  stable  organization  structure. 

The  selected  organization  will  depend  on 
many  factors.*  Some  of  the  more  important 
factors  are: 

a.  Size  of  Each  Sofrware  Development  - 
Nuafaer  of  prograiiiners  whose  output 
must  be  combined  to  make  up  one 
•working  program. 

b.  Number  of  Projects  - Few  large 
projects  (30  or  more  proarammers) 
or  many  small  projects  (10  or  less 
prograesaers)  . 

c.  Scope  of  Development  - Types  of 
woric  activity  being  performed  at 
any  one  tizie.  Are  all  programmers 
involved  in  active  development?  or 
are  some  involved  in  planning  for 
new  projects,  some  involved  in  new 
design,  and  some  involved  in  design 
maintenance? 


d.  Environment  - An  organization 
structure  must  recognize  and  be 
able  to  cope  with  the  corporate 
environment  in  which  it  exists. 

Two  important  considerations  are: 
First,  authority  deleaated  to  the 
organization  and  second,  interfaces 
with  ocher  organizations . 

Different  Structures 

Although  the  organization  of  software 
development  can  follow  one  of  the  three 
standard  foras  of  organization  structure 
(functional,  project,  matrix)  most  large 
departments  are  too  complex  to  strictly 
accossnodate  only  one  form,  thus  they  combine 
two  or  three  of  the  standard  organization 
scr\Ac  cures . 

I would  like  to  describe  ay  experience 
with  various  organization  structures  and 
how  these  structures  affect  the  process  of 
soft'ware  development. 
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Functional  Organization 

Chart  li  shows  the  structure  of  a 
functional  organization  which  has  been  used 
to  control  the  development  of  a large  soft- 
ware controlled  system. 

- Disadvantages 

a.  Tends  to  limit  the  horiznnt.il  growth 
of  prograanners . A.s  .m  ox.inplc  - in 
.1  Lriic  function.il  orpani  .*  u i.>n  few 
support  programmers' get  involved  in 
application  type  programming.  Also 
few  hardware  designers  get  deeplv 
involved  in  soffware  development. 
This  structure  is  definitely  not 
prone  to  growing  generalists. 
Functional  groups  also  have  a 
tendency  of  promoting  their  special- 
ities rather  chan  -working  toward  a 
unified  project  goal.  Project 
perspecci-/e  can  easily  get  lost. 

b.  Resolution  of  "project  type" 
technical  and  administrative 
problems  are  usually  made  by  the 
only  common  link  in  the  organization 
- Che  director.  In  many  in.scances, 
technical  problems  arise  of  such 
detailed  nature  that  the  director 
cannot  make  a proper  analysis: 
often,  the  "loudest"  or  technically 
strongest  side  wins  - to  the 
detriment  of  the  entire  project. 

c.  This  type  of  structure  is  prone  to 
instability.  If  the  director 
flounders  or  lacks  true  leadership , 
the  entire  organization  has  a 
difficult  cine  progressing.  Due  to 
Che  nature  of  its  structure,  the 

. functional  organization  cannot 
tolerate  either  overly  strong 
managers  or  an  overly  -weak  director. 

- Advantages 

a.  For  a strong  director  (hopefully 
free  from  megalomania)  this 
organization  secs  a stage  far  very 
tight  and  centralized  control. 

b.  Work  force  efficiency  and  job 
security  are  often  noted  as  the 
major  advantages  of  a functional 
organization  - "-where  people  do 
whet  they  can  do  best",  these 
advantages  arc  not  totally 
jusclfiad.  First,  progracmars 
normally  do  not  want  to  do  what 
they  can  do  bast  but  would  rather 
work  on  something  new  and 
technically  challenging.  As  far 
as  job  security  is  concamed, 
functional  organizations  have  always 
been  noted  for  their  intrinsic 
security  due  to  the  fact  chat  they 
are  not  project  oriented,  -where  the 
loss  of  project  pinpoints  a group 

of  programmers  as  being  "out  of 
•work".  I have  never  experienced 
the  advantage  of  securitv  in  a 
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functional  arganizacior. . The 
security  of  a ?rograts:er  ieoends  an 
four  factors  - in  ar.v  organization 
s true  fare . 

1.  His  work  output. 

Conduct. 

3.  His  ability  to  work  in  .nore  than 
one  discipline. 

4.  Years  with  corap.my. 


c.  Probably  the  greatest  sinale 

advantage  of  a functional  organiza- 
tion is  that  it  has  the  intrinsic 
ability  to  constantly  inprove 
sofrware  development  ttethodoloaies 
and  development  procedures.  A soft- 
ware "load  tear."  will  be  used  as  an 
exasjple.  In  a functional  organiza- 
tion this  tear  will  serve  many 
projects.  Over  a period  of  years 
the  "load  team"  members  become  very 
efficient.  In  a project  organization 
(which  will  be  described  later)  the 
load  team  is  created  only  for  that 
phase  of  a project  when  "software 
loads"  are  necessary  (certain  ...oft- 
ware  loads  iro  noi;  ncodci.1  during  the 
specification  stage  of  software 
development).  Thus,  the  sporadic- 
appearing  load  team  in  a project 
organization  neither  has  time  nor 
sufficient  experience  to  generate 
\ good  procedures  - they  seen  to  always 
) be  learning.  The  load  team  has  been 
used  as  an  example:  however,  the  same 
\ advantage  exists  in  a functional 
V organization  for  all  facets  of 
sottvare  development  from  specifi- 
cation through  design  maintenance. 

CHART  14 
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Project  Organization 

One  must  be  careful  when  looking  at  an 
organizational  str'ucture  and  assuming  that 
it  is  project  in  nature.  In  man-/  instances 
a department  which  is  considered  to  be 
project  organized  as  shown'  in  £hart  15,  is 


truly  functional  in  operation  - at  least  at 
the  supervisor  level  - which  most  critically 
determines  the  .working  environment  for 
programmers.  Chart  15  shows  that  although 
the  organization  is  "project"  when  asinz  the 
department  as  a reference,  it  if  "functional" 
when  using  the  suservisor  as  a reference 
p.oinc.  In  truth,  the  programmer  worK.s  under 
the  sane  conditions  as  if  the  entire 
department  usei  a functional  structure.  The 
.same  adv;incar,L's  .ind  Ji.sadv.-inc.iges  .is  li.scod 
for  .1  functi.'n.il  are.an  i n.i  r ! on  still  exist 

- but  now  i 111'  unit  he.iil  rcpl.icts  i ,'k'  .ii  rector 
as  the  center  of  all  m.ijor  decisions. 

An  important  conclusion  of  this  analysis 
is  that  all  structures  tend  to  look  project 
oriented  as  one  uses  higher  points  of 
reference. 

An  organization  acquires  a true  project 
structure  only  when  the  prograntter  sees  its 
effect  - and  this  exists  only  when  a 
significant  por-cion  (72")  of  an  entire 
development  is  under  the  direction  of  a 
first  level  supervisor. 

With  this  situation  the  project 
structure  acquires  the  following  advantages 
and  disadvantages: 

- Jisadv.'inc.u'cs 

a.  Project.s  mu.sc  be  kept  sm.-i  1 1 - it  is 
very  difficult  for  a suoervisor  to 
directly  control  more  than  twelve 
programmers . 

b.  Control  on  a department  level  is 
difficult  since  most  decisions  are 
made  at  low  levels. 

c.  Critical  manpower  (specialists  in 
operating  systems)  is  difficult  to 
acquire  for  each  small  project. 

d.  Training  is  high  and  costly. 

e.  Specific  design  standards  and 
procedures  either  do  not  exist  or 
are  not  uniformly  followed. 

f.  Full  "men"  must  be  made  available 
to  each  project.  Thus,  ■> 
specialist  who,  -under  a functional 
system,  would  distribute  his 
talents  among  many  projects,  must 
now  be  as.signed  to  one  project  - 
and  would  have  to  work  in  iro.a.s 
which  are  not  his  specialty  in 
order  to  remain  fully  employed. 

g.  Movement  of  people  from  one  project 
to  a more  important  project  is 
usually  more  difficult  than  in  a 
functional  structure. 

- Advantages 

a.  Decisions  are  mede  at  the  lowest 
possible  level  in  the  organization. 
Thus  allowing  quicker  decisions 
and  better  trtject  control. 
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• • b*  Since  full  responsibilicy  for  :he 

project  is  under  the  control  of  one 
supervisor,  interfaces  are  nininsized 
and  work  output  is  very  high, 
decisions  are  quickly  sade. 

c.  This  type  of  structure  tends  to 
change  specialists  into  syster. 
generalists  since  the  specialist  is 
forced  to  work  in  tiany  areas . 
Management  personnel  also  develop 
faster  in  a project  organisation 
than  in  a functional  organization. 

d.  Motivation  is  high  - most  program- 
mers would  rather  work  on  small, 
short  interval  projects,  where  they 
can  see  all  aspects  of  a single 
project. 

e.  Major  interfaces  are  keot  within 
one  group.  This  improves  project 
conmunication. 

A true  project  organization  works 
effectively  when  a department  has  responsi- 
bility for  many  independent  small  projects 
and  has  matured  to  a state  where  supervision 
is  strong  and  •technical  talent  is  abundant. 

A major  difficulty  in  directing  a large 
departaent  which  is  made  up  of  many  small 
projects  is  to  insure  a backlog  of' new 
projects  so  chat  new  work  can  constantly  be 
pushed  into  line  groups  as  old  work  is 
completed. 
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Matrix  Organization 

In  most  instances  a functional  type 
organization  is  required  to  support  large 
projects  - at  least  functional  from  a 
programmers'  viewpoint. 

A cure  for  some  of  the  problems 
encountered  in  a functional  organization  can 
be  achieved  oy  employing  a matrix  structure. 

Chart  16  shows  a diagram  of  how  a 
matrix  structure  is  put  together.  The 
standard  way  to  form  a matrix  is  to  first 
appoint  a project  manager,  give  him  no  line 
authority,  give  him  all  technical  project 
responsibility  and  then  place  a few  system 


analysts  under  his  control. 

- Disadvantages 

a.  The  Project  manager  has  a difficult 
job.  He  has  the  responsibilitv 
originally  held  by  the  director,  but 
unlike  the  director  the  project 
manager  has  no  real  authority.  It 
is  possible  to  try  to  partially 
offset  this  authority  problem  by 
telling  the  line  unit  heads  that  the 
project  manager  has  budget  control; 
but  it  soon  becomes  obvious  that 
unit  heads  mu.st  retain  responsibil- 
icy for  budgets.  The  project 
manager  can  only  complain  to  the 
director. 


b.  Matrix  organization  causes  two  major 
conflicts:  Project  manager 
conflicts  'rfich  director  since  both 
are  crying  to  perform  the  same  job. 
Project  manager  conflicts  ’rfich  line 
unit  heads  since,  in  many  cases, 
they  are  also  trying  to  perform  the 
same  job. 


- Advantages 

a.  Department  has  a built-in  technical 
cross-check.  It  should  be  noted 
chat  a similar  type  of  development 
cross-check  can  be  achieved  by 
rotating  line  unit  heads  or  line 
supervisors  (but  not  both  simul- 
taneously) at  a ci.me  which  is  early  ' 
enough  to  allow  for  project  recovery 
if  a serious  design  flaw  is 
discovered. 

b.  A project  manager  and  his  staff  are 
better  equipped  to  make  project  type 
.technical  decisions  chan  either  the 
director  or  any  single  line  unit 
head . 

c.  The  project  manager  and  hi.s  team 
form  ,1  good  pl.ico  to  .issign  sohfdiile 
I'oo  rd  I n/i  I Ion  , spec  f fi  (.•■(  I i on  control, 
.ind  spec  1.1 1 project  a emit  os.  I'hi  s 
group  c.nn  al.so  be  nssigneil  initial 
planning  for  future  new  projects. 
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Cotnrosl:e  Orzanizacian 

A flexible  organisarion  icrucrure  is 
or.e  vi-.ich  car.  cope  vic.H  boch  lar^e  ar.d  small 
sofcvara  projects;  ar.d  simultaneously  allow 
each  project  to  yxisc  at  any  stage  of  active 
development  or  design  maintenance. 

.A  structure  which  has  successfully 
worked  in  such  an  environment  exist.s  as  n 
combination  of  all  three  st.andard  org.inica- 
tion.  structures.  Functional,  project, 
matrix.  I have  called  this  structure  a 
composite  organicatior.  and  it  is  illustrated 
in  Chart  IT . 

Each  of  the  previous  organicatior. 
structures  has  certain  advantages  and 
disadvantages.  Each  structure  is  designed 
CO  be  effective  in  a specific  work  environ- 
ment. Orgar.icatior.s  developing  one  large 
sofrware  project  will  tend  to  be  functional, 
organizations  with  many  small  projects  will 
tend  to  be  project.  Organizations  with  a 
few  large  projects  will  tend  to  be  ~acrix. 
to  properly  deal  with  more  complex  environ- 
ments a composite  organization  has  been 
established. 

Important  characteristics  of  a 
composite  organization  are: 

a.  In  a manner  similar  to  the  matrix 
organization,  a project  leader  is 
appointed  to  coordinate  project 
activities  and  schedules.  Unlike 
the  matrix  organization,  the  project 
leader  is  not  resoonsible  for  total 
project  success.  Ke  is  responsible 
for  uncovering  discrepancies  and 
reporting  his  results  to  responsible 
line  management. 

b.  The  composite  organization 
recognizes  that  close  cooperation 
is  recuired  berweer.  active  design 

. groups  ar.d  "design  ma’-nter.ance" 

groups.  It  places  a single  "design 
maintenance"  super’.’isor*.*  group 
under  each  unit  head  having  new 
design  resocnsibility . This  srocess 
allows  for  efficient  movement  of 
program  responsibility  between 
supervisors  responsible  for  active 
design  and  supervisors  responsible 
for  design  maintenance. 

c.  The  composite  organization  Includes 
a separate  "new  project  group" 
responsible  for  initial  planning 

of  new  projects  and  a staff  group 
which  provides  a centralized 
service  to  all  project  leaders  and 
line  management  In  such  areas  as 
schedules . -budgets  and  computer 
ser'/ices  . 

d.  The  composite  organization  includes 
a separate  evaluation  group.  This 
group  is  used  to  provide  independent 
evaluation  of  all  software  design. 
This  gro-ao  also  maintains  svstem 
?rotot:/pes  and  supporting  computers. 
It  Is  responsible  for  configuration 


control  of  all  sofrware  that  has 
been  commercially  released  as  -well 
as  long  term  interface  with  "users" 
of  the  program. 

e.  The  composite  organization  can  act 
as  a project  organization  by 
allowing  a single  supervisor  to 
directly  control  a complete  oroject. 
For  those  projects  repuirlng  i more 
functional  arrangement,  the 
composite  scr-ucture  allows  a 
supervisor  to  directly  control  onl-/ 
the  major  portion  of  a project  and 
contract  to  the  functional  part  of 
the  compo.si'-<»  organization  for 
critical  design  assistance. 

f.  The  composite  organization  can  act 
as  a tr-ue  functional  organization 
for  projects  req-uiring  many 
different  critical  resources. 

g.  The  composite  organization 
recognizes  the  need  for  teams  and 
thereby  allows  for  team  leaders 
(other  prograntners)  to  aceuire  a 
higher  technical  stat-us  than  tear, 
members.  However,  all  team  leaders 
and  team  members  report  directly  to 
the  supervisor  for  task  assignment 
and  other  administrative  guidance. 
Team  members  consist  of  both 
prograrmners  and  technical  as.sis- 
tants.  The  function  of  an  assistant 
varies  among  groups;  however,  an 
assistant  is  usually  in-zolved  in 
documentation,  unit  testing,  editing 
and  many  software  bookkeeping 
duties. 


CHART  17 
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PROJECT  leaders  may  exist  AT  ANY  LEVEL  IN  THE 
ORGANIZATION  FOR  LARGE  PROJECTS  HE  MAY  REPORT  TO 
DIRECTOR.  Fufl  SMALL  PROJECTS  HE  MAY  REPORT  TO 
SUPERVISOR 


Conclusion 

Hanagement  of  software  develooment  is 
progressing  through  a very  rapid  stage  of 
maturity.  As  management  becomes  more 
experienced,  complex  software  projects  are 
being  developed  on  schedule  and  within 
budget.  This  paper  has  stressed  two  points. 
First,  software  developments  should  be  based 
on  historical  experience.  Freferablv  using 
exoeriencs  gained  within  the  supporting 
company.  Second,  software  developments  must 
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have  ascablished  :r.ar.aaer.enc  objectives,  a 
flexible  organization  structure  and  a pre- 
defined ieveloctient  methodology  if  manage- 
sent  is  going  to  maximize  output  with 
minimum  expenditures.  Although  this  paper 
does  not  concentrate  on  the  similarities 
between  managing  hardware  development  and 
software  development,  experience  is  showing 
that  the  same  management  principles  are 
applicable  to  both  processes. 

The  intent  of  this  paper  has  been  to 
give  an  overview  of  a few  major  areas 
associated  with  software  management.  As 
with  all  areas  of  management  chore  .nre-no 
answers  - only  appro.iehcs  - ch.it  work  .some- 
time« . ^ 
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RESOURCE  ESTL\L\TION 


Jules  Schwartz 

Computer  Sciences  Corporation 

Introduction 

The  estimation  of  resources  is  probably  the  least  precise  aspect  of  system  develop- 
ment. On  the  assumption,  however,  that  an  orderly  attempt  at  determining  resources 
is  necessary  prior  to  the  start  of  a project,  simple  guidelines  for  estimation  have 
been  developed. 

Although  these  techniques  cannot  be  relied  upon  to  produce  exact  results,  they  serve  a 
number  of  useful  purposes : 

1.  They  tend  to  produce  conservative  results,  which  is  counter  to  most 
people's  original,  if  not  final,  estimates. 

2.  They  can  be  reexamined  as  a project  proceeds,  and  better  estimates 
of  each  factor  can  be  made. 

3.  It  makes  people  think  of  many  important  ingredients  before  plunging 


ahead. 


SLIDE  1 - FACTORS  Ff'R  CONTROL  OF  ESTIMATES 
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Ingredients  for  resource  estimating  rat^e  from  clearly  quantitative  factors  for  which 
measurement  of  effect  can  be  stated  reasonably  well  to  subjective  factors  which 
require  considerable  thought  before  a numerical  contribution  to  the  estimate  can  be 
made.  Slide  1 provides  labels  for  five  types  of  estimation  parameters.  The  first 
(PROGRAM  SIZE)  is  approximate  and  can  be  derived  through  experience,  rough  design, 
prototyping,  or  the  use  of  consultants.  The  accuracy  of  this  factor  clearly  improves 
as  the  work  progresses. 

The  other  labels  represent,  from  top  to  bottom,  factors  for  which  decreasing  levels 
of  confidence  in  precise  numbers  are  possible.  Nevertheless,  the  estimate  must 
take  all  of  these  into  account. 


FACTORS  FOR  CONTROL  OF  ESTliVIATES 


• FFCGRAM  SiZE-SSTlMATE 

• STATISTICAL-MEASURED 

• OBSERVED-A.oRROXI.MATc 

• OBSERVED-JUDGMENTAL 

• OBSERVED-APPLICABLE 


SLIDE  2 - STATISTICAL 


The  statistical  factors  are  those  for  which  there  has  been  enough  experimentation  or 
experience  to  foster  reasonable  confidence  in  their  effect  on  costs. 

"A”  may  be  derived  for  a particular  organization  over  time,  utilizing  results  from 
a number  of  projects,  or  one  may  utilize  "industry  averages”  where  they  seem 
appropriate  (e.g. , 10  lines  per  day).  "A"  in  this  case  covers  the  time  period  between 
the  beginning  of  Detailed  Design  and  the  end  of  Integration. 

"C"  case  refers  to  the  difficulty  in  programming.  At  one  extreme  would  be  a batch, 
serial  process.  At  the  other  end  would  be  a real-time,  communications -oriented 
random  process. 

"P",  "L”,  and  "I"  are  all  reasonably  clear. 

Thus,  using  just  these  factors  and  System  Size(S),  the  estimate  for  required  resources 
can  be  approximated. 


ST.A'nSTICAL 


• AVERAGE  NUMBER  OF  INSTRUCTIONS  PER  DAY  (A) 

• EFFECTS  Or  COMPLEXITY  (C) 

• EFFECTS  OF  PEOPLE  (P) 

• PROGRAMMING  LANGUAGE  (L) 

• DEBUGGING  STYLE  (1) 

• REQUIRED  RESOURCES  = F (S,A,C,P,L,I,...) 
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SLIDE  3 - ESTDIATE  USING  STATISTICAL  FACTORS 

R.R,  on  this  slide  represents  the  calculation  when  only  statistical  factors  are  used 
for  Resources  Required.  The  effect  of  each  factor  is  listed  at  the  bottom  of  the  slide. 


ESTIMATE  USING  STATISTICAL  FACTORS 


rt.R.  — * p * I * I * 

A ^ 

S =r  SIZE  !N  HIGHER  LEVEL  SOURCE  LINES 
A = INSTRUCTlONS/RROGRAMIViSR/DAY 

’/2  < C < 2 
< P < 3 
1 < L < 2 
*'2  ^ I ^1 


SLIDE  4 - APPROXI^L\TIONS 


These  are  factors  ^and  other  known  situations)  which  affect  the  resources  required  for 
a system  development. 

■’ALLOWED  TIME”  influences  a project  in  the  sense  that  it  can  actually  cost  more  for 
periods  of  time  which  are  unreasonably  short.  This  is  illustrated  further  on  Slide  5, 
and  an  approximation  (M)  of  i.,s  quantitative  effect  is  given  on  Slide  8. 

Experience  (X)  with  similar  efforts  will  normally  improve  performance  considerably, 
although  unless  the  experience  is  highly  similar  it  won't  be  very  beneficial. 

The  effects  of  other  factors  will  be  explained  in  the  following  slide. 


APPROXliVlATlONS 


• ALLOWED  TIME 

• EXPERIENCE  ON  SIMILAR  SYSTEMS 


• RESOURCE  BUILDUP  OVER  TIME 


• SUPPORT  PERSONNEL 
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SLIDE  5 - TIME  RANGES 

' I 

This  is  one  author's  depiction  of  the  beneficial  effect  of  allowing  more  time  on  a pro-  | 

ject.  In  addition,  it  is  shown  that  it  is  impossible  to  complete  some  projects  within  ^ , 

certain  time  periods,  no  matter  how  much  money  is  spent.  This  is  clearly  an  approx-  ! 

imate  curve,  and  the  actual  factor  used  in  the  calculations  (M)  provides  a linear 
estimate. 


TI?vi£  RAi'^GES 


SLIDE  6 - TECHNICAL  ^L^NPOWER  RESOURCES  OVER  TIME 

This  slide  depicts  "tjpical"  growth  rates  in  necessary  technical  manpower  over  the 
lifetime  of  a system  development.  "H"  is  the  lifetime  of  a system  development.  "H" 
is  the  maximum  number  of  people  utilized  and  "E"  is  total  elapsed  time  from  the 
beginning  of  Requirements  Specification  to  the  end  of  System  Test.  The  heavily 
shaded  area  represents  the  period  during  which  the  factor  "A"  is  computed  for  this 
technique.  Along  the  abcissa,  "typical"  percentages  of  time  and  personnel  are  shown 
for  different  phases  of  a project.  These  (and  the  shape  of  the  curve)  will  vary  widely 
as  a function  of  the  parameters  utilized  for  estimating,  but  this  chart  represents  a 
reasonable  starting  point.  This  chart  doesn't  reflect  the  overlapped  nature  of  most 
projects,  in  the  sense  that  work  doesn't  stop  for  each  phase  as  neatly  as  depicted 
here.  The  growth  isn't  normally  as  much  a discontinous  step-function  as  shown  here. 
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SLIDE  7 - TOTAL  RESOURCES 

This  slide  shows  the  need  for  personnel  in  addition  to  the  technical  implementors 
discussed  earlier.  Included  are  managers,  liaison  personnel,  staff  members, 
documentors,  and  operations  personnel.  The  major  point  of  the  slide  is  to  illustrate 
the  way  in  which  the  percentage  of  "overhead”  personnel  increases  as  the  size  of  the 
technical  staff  grows.  Of  course,  this  factor  is  subject  to  variation  with  different 
organizations. 


TOTAL  RESOURCES 


FACTOR  REQUI.RED 

TECHNICAL  PERSONNEL  SUPRO.RT  PE.RSONNEL 

<12  .5 

12  — 100  1 
>100  >1 


\ SLIDE  3 - ESTIMATE  ADDING  APPROXIMATE  FACTORS 

The  expression  for  Technical  Manpower  Required  Resources  (R.R.)  is  given  in  (1). 

Included  also  are  the  expression  (M)  which  takes  elapsed  time  into  account  and  the 
limits  on  the  factor  for  experience  (X).  This  expression  includes  only  the  Statistical 
and  Approximate  Factors. 

The  expression  in  (2)  gives  the  calculation  for  the  size  of  technical  resources  during 
System  Test  in  terms  of  the  expression  computed  in  (1). 

Expressions  (3)  and  (4)  give  calculations  for  Requirements  and  General  Design 
Specification  Development. 
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SLIDE  9 - JUDGMENTAL  FACTORS 

Some  of  the  major  potential  problem  areas  which  may  arise  on  a given  project  are 
depicted  in  Slide  9.  In  the  main  they  point  out  the  costs  that  should  be  associated 
when  plans  are  made  dependent  on  unknowns  or  clearly  weak  technical  assumptions, 
including  hopes  for  acquiring  people  from  other  sources,  a willingness  or  need  to 
change  specifications  during  development,  weak  managers,  outstandingly  difficult 
technical  problems,  and  reliance  on  untried  hardware  or  software.  The  precise 
amount  to  add  to  the  cost  of  a project  for  these  situations  is  difficult  to  specify  in 
general,  but  increases  in  cost  from  10  to  30  percent  should  not  be  unusual. 


JUDGMENTAL  FACTORS 

• RELIANCE  ON  PERSONNEL  DEPARTMENT 

• DEPENDENCE  ON  OTHER  PROJECTS 

• TOLERANCE  FOR  CHANGE 

• EXPERIENCED  FAILURE 

• BREAKING  TECHNOLOGICAL  BARRIERS 

• TRUST  IN  THE  SUPPLIER 

• OTHERS 

INCREASE  NEEDED  RESOURCES  APPROPRIATELY. 
THEN.  USE  SUPPORT  FACTOR 
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SLIDE  10  - SIMPLE  OBSERVATIONS 

This  slide  illustrates  some  unverified  conclusions  about  the  conduct  of  many  system 
development  efforts.  The  main  points  are  that  most  projects  slip  to  some  extent  over 
the  time  originally  planned,  and  the  amount  slipped  (T3  on  the  Slide)  is  almost  inde- 
pendent of  the  time  originally  scheduled. 


SIMPLE  OSSERYATIO.'IS 


MINIMUM  TARGET 


MINIMIZE  T 


SLIDE  11  - MORE  OBSERVATIONS 


This  Slide  tries  to  illustrate  why  the  observations  of  Slide  9 may  be  true. 

Basically  they  occur  because  of  human  nature  and  managerial  decisions.  If  these 
curves  are  anywhere  near  accurate,  it  will  be  seen  that  the  bulk  of  the  work  doesn't 
get  done  until  near  (and  sometimes  after)  the  target  date.  This  is  probably  largely 
due  to  the  fact  that  many  plans  don't  include  much  in  the  way  of  tangible  output  until 
the  target  date. 
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SLIDE  12  - OBJECTIVE:  iLANl’  MILESTONES 


This  slide  poiats  out  the  two  valuable  features  of  providing  many  tangible  deliveries 
over  the  lifetime  of  a system  development.  The  first  advantage  is  that  the  work 
force  is  always  motivated  to  produce  results  and  thus  achieve  at  a higher  rate  through- 
out the  life  of  the  project.  The  second  advantage  is  that  each  tangible  deliverable 
milestone  provides  an  excellent  window  on  progress. 


OBJECTIVE;  iVIANY  MILESTONES 


ENERGY 
(OR  PAN(C) 


START  TARGET 

MILESTONES  = CONTROL  POINTS 
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SLIDE  13  - SUMMARY  OF  PROBLEM 


1 


This  slide  lists  some  of  the  difficulties  in  utilizing  the  preceding  techniques.  It 
is  clear  that  even  when  estimating  techniques  are  followed,  estimates  are  still  only 
rough  approximations.  However,  this  does  not  imply  that  simple  guesswork  is 
better.  Disciplined  attempts  at  recognizing  and  appreciating  the  influence  of  all 
possible  factors  i?  essential  if  a good  approximation  is  to  be  achieved. 


SUMMARY  OF  PROBLEM 

• ESTIMATES  OF  SIZE  AND  COMPLEXITY 

• NON-UNEAH  EFFECTS 

• INTERACTIONS  AMONG  VARIABLES 

• SPECIAL  CIRCUMSTANCES  — OTHER  FACTORS 

• STATISTICS 

• EFFECTS  OF  NEW  CONCEPTS  (D.P.,  S.P.,  D.B.  . . .) 

• MOTIVATIONAL  FACTORS 

BUT  DISCIPLINE  IN  ESTIMATION  IS  VALUABLE. 
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The  Cost  of  Developing  Large-Scale  Software* 


RAY  W.  WOLVERTON,  member,  ieee 


Abstract— Th«  work  of  softworo  cost  forecastiiic  falls  into  two 
parts.  First  wo  make  what  we  call  structural  forecasts,  and  then  we 
calculate  the  absolute  dollar-volume  forecasts.  Structural  forecasts 
describe  the  tschnolofr  snd  function  of  a software  project,  but  not 
its  sixe.  We  allocate  resources  (costs)  over  the  project's  life  crele 
from  the  structural  forecasts.  Judcment,  technical  knowledge,  and 
econometric  research  should  combine  in  making  the  structural  fore- 
casts. A methodology  based  on  a 25  X 7 structural  forecast  matrix 
that  has  been  used  by  TRW  srith  good  results  over  the  past  few 
years  is  presented  in  this  paper.  With  the  structural  forecast  in  hand, 
we  go  on  to  calculate  the  absolute  doUar-eolume  forecasts.  The 
general  logic  followed  in  “absolute”  cost  estimating  can  be  based 
on  either  a mental  procasa  or  an  explicit  algorithm.  A cost  estimating 
algorithm  is  presented  and  fire  traditioa  methods  of  software  cost 
forecasting  are  described:  top-down  estimating,  similarities  and 
differences  estimating,  ratio  estimating,  standards  estimating,  and 
bottom-up  estimating.  All  forecasting  methods  suffer  from  tiie  need 
for  a valid  cost  data  base  for  many  estimating  sitttations.  Software 
information  elements  that  experience  has  shown  to  be  useful  in 
establishing  such  a data  base  are  given  in  the  body  of  the  paper. 
Major  pricing  pitfalls  are  identified.  Two  case  studies  sre  presented 
that  illustrate  the  software  cost  forecasting  methodology  and  his- 
torical results.  Topics  for  further  work  and  study  are  suggested. 

Index  Terms— Computer  programmer  code  productivity  rates,  cost 
data  base  used  in  cost  estimation  nMchanism,  cost  of  developing 
custom  software,  incentive  fee  structure  influence  on  computer  soft- 
ware development,  principles  of  priciag  computer  software  proposals, 
resource  allocation  in  computer  program  development,  software 
cost  estimating  methods  and  general  logic,  software  cost  estimation 
algorithm,  software  development  and  test  life  cycle— cost  impact, 
systems  approach  to  pricing  large-scale  software  projects. 

« 

INTRODUCTION 

ESTIM.VTING  the  cost  of  a large-scale  computer 
program  ha.s  traditionally  been  a risky  undertaking. 
Now  that  software  has  been  with  us  for  nearly  two 
decades,  it  is  reasonable  to  hope  that  we  have  learned 
something  that  would  make  our  predictions  less  un- 
reliable. .\t  TRW  we  have  been  motivated  to  make  a 
serious  effort  toward  improving  our  software  estimating 
methods  for  large-scale  software  systems,  for  a set  of  very 
compelling  reasons. 

1)  Our  customers  have  shown  a growing  unwillingness  to 
accept  cost  and  schedule  overruns  unless  the  penalties 
were  increasingly  borne  by  the  software  developer. 

■J ) Partly  as  a consequence  of  1 ) , we  have  entered  into 
s<jftware  development  contracts  where  both  adherence  to 
predicted  costs  and  on-time  deliveiy  were  incentivixed. 

The  author  is  with  the  TRW  Systems  Group,  Redondo  Beach, 
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That  means  we  made  more  money  if  we  could  predict 
accurately  the  cost  and  the  time  it  would  take  to  do  the 
job. 

3)  We  found  that  we  could  improve  our  estimates  only 
by  improving  our  understanding  of  exactly  what  steps  and 
processes  were  involved  in  software  development,  and  this 
understanding  enabled  us  to  manage  the  effort  better.  The 
better  management,  in  turn,  improved  our  estimates. 

This  paper  presents  the  essential  results  of  our  efforts 
to  improve  our  software  cost  estimating  techniques.  It 
should  be  understood  that  the  specific  contracts  that  are 
used  here  were  government  contracts  whose  particular 
provisions  shaped  those  efforts  to  some  degree.  Neverthe- 
less, we  feel  that  much  of  what  we  learned  is  of  general 
interest  and  can  be  applied  or  adapted  to  nearly  any  soft- 
ware development  program  of  any  size. 

We  can  start  with  a review  of  some  of  the  ways  in  which 
software  development  differs  from  hardware  development, 
and  which  have  traditionally  contributed  to  the  problem 
of  estimating  software  costs.  One  of  these  ways  is  the 
problem  of  managing  the  people.  The  nature  of  program- 
mers is  .such  that  interesting  work  gets  done  at  the  ex- 
pense of  dull  work,  and  documentation  is  dull  work. 
Doing  the  job  in  a clever  way  tends  to  be  a more  important 
consideration  than  getting  it  done  adequately,  on  time,  and 
at  reasonable  cost.  Programmers  tend  to  be  optimistic, 
not  realistic,  and  their  time  estimates  for  task  completion 
reflect  this  tendency.  The  software  engineer,  the  mathe- 
matician, and  the  programmer  have  trouble  communi- 
cating. Visible  signs  of  programming  progress  are  almost 
totally  lacking. 

.\nother  set  of  difficulties  arises  from  the  nature  of  the 
product.  There  are  virtually  no  objective  standards  or 
measures  by  which  to  evaluate  the  progress  of  computer 
program  development.  Software  characteristics  are  often 
in  conflict,  requiring  frequent  tradeoffs  among  such 
factors  as  core  storage  requirement  verstis  tight  code,  fast 
execution  time,  external  storage  to  minimize  I/O  time, 
maintainability  by  the  eventual  user,  flexibility  or  adapt- 
ability to  new  needs,  accuracy  and  reliability,  and  self- 
check  and  fail-safe  modes  of  operation.  The  applications 
software  developer  finds  himself  in  a dynamically  evolving 
and  constantly  changing  environment. 

Softw'are  planning  has  problems  stemming  from  the 
natural  desire  to  get  going,  which  means  taking  shortcuts. 
What  are  the  steps  that  should  precede  the  start  of  coding, 
and  how  does  the  knowledgeable  manager  allocate  his 
limited  resources?  Planning  also  is  made  difficult  by  the 
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problem  of  translating  the  customer’s  functional  require- 
ments into  software  requirements,  and  later  defining  the 
procedures  by  which  you  determine  that  those  original 
customer  requirements  have  been  met.  It  is  not  always 
easy  to  determine  when  you  are  through  and  whether  you 
have  delivered  what  you  said  you  were  going  to  deliver. 

The  overall  approach  we  have  developed  for  dealing 
with  these  problems  and  producing  reasonably  reliable 
cost  estimates  for  large-scale  scjftware  systems  consists  of 
the  following  basic  elements,  which  will  be  discussed  in 
detail  in  the  remainder  of  this  paper. 

1)  Understanding  exactly  what  the  software  develop- 
ment process  consists  of  over  its  life  cycle. 

2)  Recognizing  the  pitfalls  in  the  pricing  of  software. 

.3)  Establishing  and  maintaining  a good  data  base 

reflecting  our  history  of  actual  software  development 
costs. 

4)  Determining  the  most  cost-effective  allocation  of 
resources  over  the  entire  period  of  the  expected  contract. 

THE  SOFTWARE  DEVELOPMENT  CYCLE 

Because  software  is  during  most  of  its  development 
cycle  (and  even  afterward)  a basically  intangible  product, 
it  is  even  harder  to  control  than  a complex  hardware 
system.  We  have  learned  by  trial  and  error  that  no  cost 
predictions  can  be  fulfilled  unless  the  mechanism  for 
management  control  is  solved  in  advance.  We  have 
established  five  management  principles  that  apply 
throughout  the  software  development  cycle  to  reduce  the 
oroblem  of  control  to  manageable  size. 

1)  Produce  software  documentation  that  meets  estab- 
lished programming  standards,  is  accurate,  is  produced 
according  to  the  needs  of  the  audience  who  will  use  it, 
and,  is  most  of  all,  an  instrument  by  which  management 
controls  the  project, 

2)  Conduct  technical  reviews  against  predetermined 
acceptance  criteria  to  the  end  that,  upon  customer  ap- 
proval. predetermined  baselines  are  established. 

3)  Control  software  physical  media  (tapes,  disks, 
decks)  to  assure  use  of  a known  configuration  in  testing, 
demonstration,  certification,  and  delivery. 

4)  Apply  software  configuration  management  controls 
and  procedures  to  assure  that  required  changes  to  base- 
lines are  implemented,  tested,  fully  documented,  and 
understood. 

5)  Provide  a data  reporting,  repository,  and  control 
system  to  assure  that  all  problems,  deficiencies,  change 
proposals,  and  software  configuration  data  are  analyzed, 
reported,  recoverable,  and  available  to  all  project  and 
user  personnel  when  and  where  needed. 

Technical  reviews  are  required  during  the  software 
development  life  cycle.  Reviews  ensure  that  the  resulting 
software  products  meet  the  requirements  established  for 
that  level  of  completeness  and  understanding,  A “base- 
line” is  established  at  designated  points  in  time  during  the 
software  development  cycle  when  the  requirements  (or 


design)  have  been  defined,  documented,  reviewed,  and 
approved.  From  that  time  fonvard,  that  baseline  serves 
as  the  reference  point  for  evaluating  and  managing  any 
changes  in  scope,  price,  or  .schedule. 

In  the  specific  cases  to  be  discussed,  certain  of  these 
principles  were  incorporated  in  government  software 
procurement  policies  applied  to  the  contract,  but  no 
manager  of  a large  software  development  should  ignore 
them  whether  they  are  required  by  his  customer  or  not. 
For  example,  without  software  configuration  manage- 
ment— configuration  identification,  configuration  control, 
configuration  status  accounting,  and  verification — the 
software  development  manager  will  spend  much  of  his 
time  solving  problems  that  need  not  have  arisen.  Soft- 
ware is  controlled  through  control  of  documentation 
within  a structured  software  development  process. 

There  are  several  ways  of  analyzing  the  software  de- 
velopment process,  but  the  one  we  have  chosen  leads  to 
the  definition  of  seven  phases,  or  steps,  each  ending  in  a 
discrete  event  or  docviment  (see  Fig.  1).  The  steps  are 
basically  sequential,  but  in  practice  there  are  iterations 
between  adjacent  steps  and  even  between  any  of  the 
steps.  The  seven  steps  are  as  follows. 

Slep  1 — Performance  and  Design  Requirements:  This 
document  establishes  the  requirements  for  performance, 
design,  test,  and  qualification  of  the  software  at  the  system 
level. 

Step  2 — Implementation  Concept  and  Test  Plan:  This 
document  provides,  at  an  early  date,  the  preliminary 
design  concepts  that  meet  the  requirements,  and  identifies 
the  preferred  approaches,  design  tradeoffs,  and  alterna- 
tives, and  leaves  no  high-risk  technology  area  unresolved. 

Step  3 — Interface  and  Data  Requirements  Specification: 
This  document  defines  the  interfaces  between  subsystems 
and  major  elements  within  subsystems,  including  pres- 
entation of  table  and  file  structure,  data  origin,  desti- 
nation. and  update  characteristics. 

Step  4 — Detailed  Design  Specification:  This  document 
provides,  in  precise  detail,  the  complete  design  and  ac- 
ceptance specifications  for  the  softw'are  at  the  routine 
level,  which  includes  detailed  flow  charts,  equations,  and 
logic.  It  is  the  source  material  from  which  the  program  is 
coded. 

Step  5 — Coding  and  D^nigging:  Upon  des^  review 
and  approval  of  the  Step  4 documentation  package, 
which  may  be  staggered  in  time  and  organized  into 
volumes  dealing  with  a single  computer  program  module, 
the  coding  of  the  software  can  begin.  This  step  is  devoted 
to  coding  and  debugging  (and  amending  documentation) , 
using  debugging  aids  and  desk  check  of  the  first  compi- 
lations. The  phase  terminates  for  a given  module  when  it 
has  passed  the  development-level  verification  tests  im- 
posed by  the  designer,  and  he  turns  the  module  over  for 
assembly  onto  the  master  tape  for  the  beginning  of  in- 
dependent system  validation  testing. 

Step  6 — System  Validation  Testing:  As  the  software 
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Fig.  1.  Typical  custom  software  development  and  test  steps,  showing  seven  unique  documents. 
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modules  are  released  by  the  design  group  for  assembly 
onto  the  master  tape,  the  buildup  of  the  software  package 
into  successively  more  complex  interfaces  and  capabilities 
can  commence.  When  two  or  more  modules  are  assembled, 
the  first  stages  of  “system  testing”  can  get  underway 
according  to  the  test  plan  started  at  Step  2 and  upgraded 
in  level  of  detail  at  Step  4 to  include  test  procedures  and 
quantitative  acceptance  criteria.  The  phase  terminates 
when  all  modules  have  been  successfully  tested  against  the 
predetermined  acceptance  criteria  at  the  system  level. 

Step  7 — Certificaiion  and  Acceptance  Demonstralion: 
This  final  step  is  devoted  to  formal  acceptance  of  the  soft- 
ware system  by  subjecting  it  to  previously  defined  ac- 
ceptance and  test  specification  procedures,  which  are 
executed  in  as  near  an  operational  environment  and  host 
computer  configuration  as  possible.  The  phase  terminates 
with  successful  “operational  demonstration,”  witnessed 
by  the  customer  and  his  quality  assurance  team.  The 
software  packages  arc  then  ready  for  installation,  integra- 
tion, and  checkout  in  the  operational  environment. 

The  final  step  in  the  software  development  life  cycle  is 
the  forerunner  of  the  operations  and  naaintenance  phase, 
strictly  speaking,  Step  8.  This  phase  is  concerned  with 
installation,  integration,  and  checkout  in  the  full  oper- 


ational configuration.  The  software  contractor  as*’'sts  in 
any  way  required  to  integrate  the  software  products  of  the 
development,  or  acqusition,  phase  into  an  operational 
software  package  that  meets  the  original  (and  updated) 
performance  and  design  requirements,  the  product  of 
Step  1 and  subsequent  updates.  The  activities  of  this 
phase  will  also  include  training,  rehearsal  support,  software 
problem  reporting,  fault  isolation  and  correction,  formal 
problem  closure  and  documentation  update,  and  finally, 
first  mission  operations  support.  Because  this  is  not  part 
of  the  software  development  effort,  we  cost  it  differently — 
usually  as  a level  of  effort. 

PRICING  PITFALLS 

Pricing  is  the  complete  process  by  which  we  arrive  at  a 
price  that  we  quote  to  the  customer.  Cost  estimating  is  a 
major  part  of  that  process,  although  it  frequently  has  to 
be  done  several  times  before  a final  price  is  arrived  at. 
The  general  principles  involved  m pricing  large  RAD 
efforts  of  any  kind  have  been  defined  by  Beveridge  [1], 
and  apply  to  large  software  developments  as  well. 
Beveridge’s  discussion  centers  around  the  preparation  of 
a proposal  in  response  to  a government  Request  for 
Propoul  (RFP),  but  his  common-sense  principles  are 
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applicable  to  any  complex  pricing  exercise  where  many 
individuals  are  involved. 

There  is  a general  tendency  on  the  part  of  designers  to 
gold-plate  their  individual  parts  of  any  system,  but  in  the 
case  of  software  the  tendency  is  both  stronger  and  more 
difficult  to  control  than  in  the  case  of  hardware.  .V  major 
task  of  management  is  to  make  certain  that  the  design 
being  propo.sed  (and  pricedl  does  meet  the  customer’s 
need,  and  that  each  individual  componimt  d(>sign  can  be 
traced  to  a specific  need,  while  at  the  .same  time  it  does  not 
provide  more  ( more  speed,  more  accuracy,  leas  need  for 
core,  etc.)  than  the  customer  needs  or  wants.  Anything 
being  offered  a customer  beyond  what  he  has  asked  for 
should  be  clearly  identified  (and  priced)  as  an  option. 
That  is  what  is  meant  by  the  “fixed-price  attitude”; 
what  is  the  absolute  minimum  I can  do  to  satisfy  the 
needs  stated  in  the  RFP? 

SOFTWARE  COST  ESTTMATIXG  .METHODS 

The  software  industry  is  young,  growing,  and  marked 
by  rapid  change  in  technology  and  application.  It  is  not 
surprising,  then,  that  the  ability  to  estimate  costs  is  still 
relatively  undeveloped.  Even  though  many  organizations 
are  tiying  to  devise  more  scientific  and  objective  means 
for  estimating  costs  of  large  software  .systems,  the  present 
state  of  the  art  is  largely  judgmental.  We  will,  how- 
ever, discuss  certain  important  exceptions  shortly.  A 
review  of  other  firms  and  agencies  confirm  that  we  are 
facing  a problem  common  to  all  3-Col- 

Estimating  the  cost  of  producing  computer  software 
relies  heavily  on  the  judgment  of  experienced  performers. 
The  software  analyst,  or  estimator,  normally  breaks  the 
total  job  into  elements  that  are  estimated  separately  and 
then  summarized  into  an  estimate  for  the  total  job.  The 
estimating  analysis  and  synthesis  may  appear  as  a mental 
process  or  may  involve  an  explicit  algorithm  CfiJ 

In  either  case,  an  empirical  data  base  is  ased  as  an  ob- 
jective reference,  and  the  estimator  uses  his  judgment  to 
account  for  differences.  Pieces  of  information  used  in  the 
comparisons  and  adjustments  for  differences  include: 
a)  analysis  of  initial  requirements;  b)  allocation  of  re- 
quirements to  software  modules;  c)  estimates  of  number 
of  object  instructions  per  module;  d)  complexity  and 
technological  risk ; e)  user  environment  and  characteristics 
of  the  customer;  f)  computer  of  choice  and  interchange- 
ability  among  other  user  sites;  g)  higher  order  language 
of  choice  or  criteria  for  use  of  assembly  language;  h)  type 
of  software  to  be  developed;  i)  whether  software  is  to  be 
delivered  to  an  operational  user;  j)  technical  experience 
on  that  type  of  job;  k)  capabilities  of  the  member  of  the 
technical  staff  who  probably  will  do  the  work;  1)  type  of 
fee  and  its  incentive  structure;  m)  length  of  deve'opment 
time;  n)  single-model  multiple-release  versus  multiple- 
model  single-release  development  concept;  o)  performance 
record  of  other  large-scale  systonui  (.AWAC,  SAGE,  etc.) 
in  the  number  of  instructions  and  development  man- 


months;  and  p)  management  factors  to  do  with  pro- 
ductivity rates,  error  rates,  work  environment,  availability 
of  computer  time,  and  many  other  variables. 

Most  estimators  u.se  a logical  sequence  in  establishing 
their  estimates.  The  logic  uses  exchange  coefficients  be- 
tween some  measurable  parameter  and  its  cost  and 
adjustment  factors  derived  from  experience.  One  of  the 
pitfalls  is  that  estimating  ratios  .should  be  directly  trace- 
able to  recorded  cost  facts  (not  hearsay),  and  even  the 
factual  data  can  contain  varnng  and  unstated  allowances 
for  risk  H'sy  have  assumed  a 40-h  work  week,  our 
records  show  we  charged  that  rate,  but  our  records  do 
not  show  that  most  personnel  worked  4.O-.50  h/week  or 
more).  Our  long-term  objective  at  TRW  has  been  to 
create  a systematic  software  development  estimation 
system  that  can  significantly  reduce  statistical  variances 
between  estimated  costs  and  actual  costs,  and  is  not  only 
accurate  in  that  sense  but  also  simple,  fast,  and  con- 
venient. We  will  briefly  look  at  estimation  methods  in 
general,  and  two  in  particular  that  have  been  developed 
and  reduced  to  practice  at  TRW. 

General  Logic  F oUoued  in  Estimating 

Traditional  cost  estimating  procedures  start  with 
fixing  the  size  of  each  activity,  its  start  date,  and  duration. 
When  necessary,  adjustments  are  made  to  account  for  the 
caliber  of  performer  personnel  to  be  assigned,  risk,  com- 
plexity, uncertainties  in  requirements,  and  so  on.  Finally, 
the  amount  and  type  of  manpower  (man-months  per 
month)  and  computing  resources  (hours  {jer  month)  are 
converted  to  dollar  costs  by  applying  bid  rates.  Other 
direct  charges  (documentation  costs,  travel,  etc.)  are 
added,  and  summaries  are  made  through  the  pricing 
system.  Traditional  methods  can  be  classified  as  one  or 
more  of  the  techniques  described  below, 

/)  Top-Dmvn  Estimating:  The  estimator  relies  on  the 
total  cost  or  the  cost  of  large  portions  of  previous  projects 
that  have  been  completed  to  estimate  the  cost  of  all  or 
large  portions  of  the  project  to  be  estimated.  History 
coupled  with  informed  opinion  (or  intuition)  is  used  to 
allocate  costs  between  packages.  .Among  its  many  pitfalls 
is  the  substantial  risk  of  overlooking  .special  or  difficult 
technical  problems  that  may  be  buriinl  in  the  project 
tasks,  and  the  lack  of  details  needed  for  cost  justification. 

i)  Similarities  and  Differences  Estimating:  The  esti- 
mator breaks  dowm  the  jobs  to  be  accomplished  to  a level 
of  detail  where  the  similarities  to  and  differences  from 
previous  projects  are  most  evident.  Work  units  that  cannot 
be  compared  are  estimated  separately  by  some  other 
method. 

,3)  Ratio  Estimating:  The  estimator  relies  on  sensitivity 
coefficients  or  exchange  ratios  that  are  invariant  (within 
limits)  to  the  details  of  design.  The  softwiire  analyst 
estimates  the  size  of  a module  by  its  number  of  object 
instructions,  classifies  it  by  type,  and  evaluates  its  relative 
complexity.  .An  appropriate  cost  matrix  is  constructed 
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fn>m  a cost  data  base  in  terms  of  cost  per  instruction,  for 
that  type  of  software,  at  that  relative  complexity  level. 
Other  ratios,  empirically  deriv'ed.  can  be  used  in  the  total 
estimation  process,  for  instance,  computer  usa(?e  rate 
basetl  on  central  processire  unit  iCPU'  time  per  in- 
struction. peripheral  usaire  to  CPI*  usage,  engineers  per 
secretary,  and  so  forth.  The  method  is  simple,  fast,  con- 
venient. and  useful  in  the  proposal  tmvironment  and 
beyond.  It  .suffers,  as  do  all  methods,  from  the  need  for  a 
valid  cost  data  base  for  ntany  estimating  situations 
( businf'ss  versus  scientific,  real-time  versus  nonreal-dme, 
operational  versus  nonoperational'. 

4'  SlniidarHs  Estiwatimi:  The  estimator  relies  on 
standards  of  performance  that  have  been  systematically 
develops!.  These  standards  then  become  stable  reference 
points  from  which  new  tasks  can  be  calibrated.  Many 
mature  industries,  such  as  manufacturing  and  construc- 
tion. use  this  method  routinely.  The  method  is  accurate 
onl>-  when  the  same  operations  have  been  performed 
repeatiMily  and  good  records  are  availhble.  The  pitfall  is 
that  custom  software  development  is  not  “performed 
repeatedly.’’ 

.5)  Bottom-l'p  EstimeUintj:  This  is  the  teclinique  most 
commonl>-  used  in  estimating  government  research  and 
development  contracts.  The  total  job  is  broken  down  into 
relatively  small  work  packages  and  work  units.  The  work 
breakdown  is  continued  until  it  is  reasonably  clear  what 
steps  and  talents  are  involved  in  doing  each  task.  Each 
task  is  then  estimated  and  the  costs  are  pyramided  to 
form  the  total  project  cost.  -An  advantage  of  this  technique 
is  that  the  job  of  estimating  can  be  distributed  to  the 
people  who  will  do  the  work.  A difficulty  is  the  lack  of 
immediate  perspective  of  the  most  important  parameter 
of  all;  the  total  cost  of  the  project.  Iii  doing  detailed 
estimates,  the  estimator  is  not  sensitive  to  the  reasonable- 
ness of  the  total  cost  of  the  software  package.  Therefore, 
top-down  estimation  is  used  as  a check  on  the  bottom-up 
method. 

The  estimation  process  in  nearly  always  a combination 
of  two  or  more  of  the  basic  classifications  given  above. 
We  will  briefly  discuss  two  methods  that  have  been  used 
in  estimating  the  cost  of  developing  large-scale  software 
at  TRW;  a method  we  used  in  June  1969  which  we  will 
call  “snuxithing  and  extrapolation,”  and  the  current 
methml  we  used  beginning  in  October  1970,  which  we  will 
call  “a  software  cost  estimation  algorithm.” 

.1  Syvtenia  Apprixich  lo  ^ioflware  Coat  Ealimation 

To  b«‘  as  free  as  poasible  to  deal  with  the  cost  estimating 
methiKl.  we  will  use  hypothetical  or  normalized  numbers 
wheii>  numbers  are  m-cessary.  Assume  the  software 
development  and  ti*st  life  cycle  consists  of  the  seven  steps 
previously  defined,  except  that  the  starting  point  at 
contract  go-ahead  is  immediately  on  completion  of  Step  2. 
We  need  to  understand  this  assumption  clearly,  since  it 
establishes  the  remaining-milcstone  events  which  are  in- 


cluded in  the  cost  which  was  negotiated  with  the  customer. 
In  other  words,  the  proposal  was  at  a level  of  detail  such 
that  it  was,  by  itself,  the  implementation  concept  and 
test  plan.  Step  2.  The  RFP  package,  together  with  the 
bidder’s  briefing,  was  very'  detailed,  technically  complete, 
and  was.  by  itself,  the  performance  and  design  require- 
ments. Step  1. 

Our  reference  project  for  the  most  recent  past  successful 
.software  project  was  identified,  and  its  case  history  was 
laid  out  in  terms  of  the  actual  events,  costs,  start  dates, 
and  durations.  The  normalized  contract  cost  as  a function 
of  the  normalized  period  of  performance  is  shown  in  Fig.  2. 
We  will  call  this  case  history  .4.  The  cost  distribution  by 
development  activity,  that  is,  the  actions  that  characterize 
the  intervab  between  discrete  milestone  events,  is  showm 
in  Fig.  .3.  Using  case  history  ,4  we  can  identify  similarities 
and  differences  with  respect  to  our  cost  estimating  for 
our  new  project.  The  basic  cost  method,  smoothing  and 
extrapolation,  is  a combination  of  earlier  methods  plus 
some  new  ideas  for  the  new  project.  We  will  call  the  new 
project  case  history’  B. 

A major  development  cycle  difference  was  that  case  .4 
has  a baseline  design  specification  deliverable  under 
contract,  which  accounted  for  about  12  percent  of  the 
total  cost  as  shown  by  “analysis.”  and  also  had  the 
implementation  concept  and  test  plan  deliverable  under 
contract,  which  accounted  for  another  18  percent  as  showm 
by  “design”  (see  Fig.  31.  Even  though  the  software  de- 
velopment cycle  was  different,  we  had  exact  data  on  the 
manpower  assigned  to  the  balance  of  the  milestone 
events,  which  are  similar.  We  now  are  in  a position  to  ex- 
trapolate from  case  .4  to  case  B.  Furthermore,  since  case 
.4  had  been  produced  on  time  and  under  cost,  we  were  in 
a strong  position  for  proving  demonstrated  cost  per- 
formance on  a previous  project. 

.A.  major  functional  difference  was  that  case  B repre- 
sented a unified  communication,  command,  and  control 
software  system,  whereas  case  A did  not  have  the  com- 
mand function.  Immediately,  greater  attention  was  given 
to  the  analysis  and  design  of  that  new  technology  area 
and  its  interfaces.  Even  here,  important  similarities  w’ere 
identified  in  the  history  file  because  in  case  ,4  the  “similar” 
command  function  was  the  responsibility  of  another 
associate  contractor.  We  concentrated  on  improving  the 
information  transfer  by  having  full  control  of  the  inter- 
face, and  by  approaching  the  solution  from  an  entirely  new 
direction.  We  decided  the  technical  risk  was  somewhat 
higher  for  case  B because  our  cost  data  base  was  in- 
complete, and  because  the  command  algorithm  design, 
coding,  and  testing  were  innovative.  We  developed  the 
overall  system  block  diagram,  and  allocated  statement  of 
work  tasks  to  software  functions.  We  were  in  a position 
for  a first  rough  cut  target  cost. 

The  whole  idea  in  this  case  was  a)  to  price  the  total 
software  system  from  the  top  down  using  the  experience 
of  the  system  concept  group,  b)  to  price  each  work 
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Fi«.  3.  Cost  distribution  by  activity  during  full  period  of  per- 
formance (case  history  A),  All  activities  include  documentation 
and  travel  costs. 


package  from  the  bottom  up  using  the  oxperience  of  each 
of  the  five  work  package  managers,  and  o to  conduct  a 
mock  negotiation  between  the  system  conct'pt  group  and 
each  work  package  group  taken  one  at  a time.  Differences 
were  derived,  costs  were  traced  back  to  source  authority 
requirements  in  the  customer’s  statement  of  work,  and 
adjustments  made  between  the  cost,  probable  risk,  and 
scope  of  work. 

In  both  methods  the  basic  unit  of  estinaation  was  the 
“man-month"  for  a given  task  or  element.  The  distribution 
of  the  work  ( man-months  per  month)  was  driven  by  two 
critical  variables;  initial  operational  capability  (IOC)  and 
full  operational  capability  (h’OC)  software  configurations 
were  required  by  the  RFP,  and  a new  incentive  fee  struc- 
ture was  required  by  the  RI'P.  Each  will  be  briefly 
addressed  by  assessing  iu  impact  on  the  cost  method. 

The  distribution,  or  manpower  spread  over  the  proposed 
period  of  performance,  was  initially  determined  by  the 
system  concept  group  using  one  new  idea.  That  idea  was 
first  to  assume  that  the  total  development  manpower 
resource  was  1000  man-months,  and  then  to  determine  the 
shape  of  the  manpower  distributiun  over  tin#  that 
maximized  the  fee  incentive  with  least  risk,  such  that  the 


area  under  the  curve  remained  constant  (namely,  1000 
man-months).  This  curve  would  serve  to  generate  the 
master  schedule  for  IOC  and  FOC  events.  The  amplitude 
of  the  curve  would  be  the  primary  focus  in  the  .separate 
negotiations  with  the  five  work  package  managers. 

The  amplitude  represented  the  number  of  man-months 
per  month  for  any  given  time  slice.  The  sum  of  the 
manpower  estimates  of  each  of  the  five  work  packages  at 
any  time  point  determined  that  month’s  manpower 
requirements,  and  the  sum  of  those  over  all  time  de- 
termined the  total  IOC, 'FOC  manpower  requirements, 
hence  total  cost.  Of  the  five  initial  estimates,  four  were 
judged  too  high  in  the  internal  first  cut  negotiations,  and 
one  was  judged  too  low. 

'The  sliape  of  the  first  and  second  estimate  by  the 
system  concept  group  is  given  in  Fig.  4.  It  is  a poor  distri- 
bution with  steep  manning  rates,  high  peaks,  and  sharp 
declines.  Con.siderable  smoothing  was  required  to  achieve 
a unimodal  increasing  (and  decreasing)  shape,  at  com- 
fortable manning  rates  and  reasonable  levels  and  still 
meet  the  original  goal  we  set  with  respect  to  incentive 
fee.  The  shape  was  drastically  altered  by  introducing  the 
concept  of  “staggered  milestone  events."  That  is,  instead 
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Fir.  4.  (a)  Manpower  estimate  with  Mngle>relea80  milestone  approach,  showing  key  deliverables  and  labor  alloca* 
tion  of  1000  man-months.  <b)  Manpower  estimate  with  single-release  milestone  approach,  showing  uneven  labor 
spread  of  1000  man-months. 

j(  on<*  event  4 * and  une  event  { steps  as  previously  system  level  testing  by  the  independent  test  and  valida- 
defined),  we  planned  for  a total  of  six  each.  We  could  do  tion  group  could  commence  soon  enough  to  meet  all 
this  because  of  modular  construction  of  the  software  to  incentives^  at  least  by  plan.  The  smoothed  shape  of  the 
the  end  that  each  module  was  ready  when  needed,  and  third  estimate,  and  the  one  that  influenced  the  time 
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Fig.  .y  (a)  Manpower  estimate  with  staggered-release  milestone  approach,  showing  inrremental  deliver}-  of  detailed 
design  specifications,  (b)  Manpower  estimate  with  staggered-release  milestone  approach,  showing  '“smoothed" 
labor  spread  of  1000  man-months. 


placement  of  the  incentive  events,  is  shown  in  Fig.  ,5. 
Both  sets  of  curves  show  only  IOC  activities  for  simplicity; 
as  IOC  man  loading  phased  down.  FOC  phased  up,  in 
general. 

A simplified  description  of  the  fee  structure  that  in- 
fluenced the  shape  of  Fig.  .j  is  that  cost,  schedule,  and  per- 


formance were  incentiviied  (from  a maximum  of  l.i  ptr- 
cent  of  final  target  cost  to  a minimum  of  0)  based  on 
certain  predetermined  rules  that  would  be  applied  at  three 
discrete  milestones;  Step  A),  the  detailed  demgn  speci- 
fication, Step  o),  the  updated  detailed  design  specifica- 
tion with  computer  program  listing,  and  Step  7),  opera- 
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Fig.  6.  Incentive  fee  structure  (case  history  B). 


tional  demonstration  (OD).  The  plan  called  for  three 
IOC  and  three  FOC  dates. 

The  effect  of  a two-step  procurement,  such  as  an 
IOC  followed  by  a later  FOC,  is  as  follows  [7],  The 
IOC  milestone  events  and  OD  constitute  the  only  way 
that  any  fee  can  be  earned,  in  discrete  steps  as  shown  in 
Fig.  t).  The  earned  fee  percentages  apply  to  the  target 
cost  for  the  total  lOC/FOC  task.  The  incentive  fee 
structure  provides  for  fee  penalties  for  failure  to  meet  the 
contract  dates  for  satisfactoiy  demonstration  of  milestone 
steps  4),  .5),  and  the  OD.  These  penalties  are  assessed 
as  shown  in  Fig.  7,  on  a linearly  graduated  basis  in  units 
of  whole  days  late,  where  “days  late”  mean  calendar  days 
from  the  date  specified  in  the  contract.  The  penalties 
apply  separately  to  both  IOC  and  FOC.  That  is,  the 
maximum  penalty  for  lateness  in  demonstrating  the  OD 
of  the  IOC  is  l."»  percent;  it  is  also  1.5  percent  for  failure 
to  demonstrate  the  OD  of  the  Ff)C.  This  insures  priority 
to  the  IOC,  but  also  insures  responsible  attention  to  the 
FOC. 

To  be  effective,  the  basic  incentive  structure  must  be 
simple,  even  though  it  is  necessary  in  the  contract  to 
address  the  major  contingencies  and  allowable  options. 
If  the  basic  incentive  structure  is  not  simple  it  will  not 
readily  be  grasped  by  the  many  people  at  all  levels  of  the 
contractor’s  plant  whose  work  affects  the  chance  of  success. 
If  they  do  not  understand  it,  they  will  not  do  anything 
diffcri'ntly  because  of  the  incentive  structure.  If  they  do 
not.  the  incentive  contract  will  have  failed  to  achieve 
its  fundamental  purpose:  the  people  who  work  on  all 
aspects  of  the  entire  undertaking  must  be  conscious  of  the 
incentive  and  must  do  their  work  with  more  care  and 
quality  because  of  it. 

The  entire  incentive  approach  presumes  that  the 
contractor  will  take  specific  internal  implementing  action. 
It  should  include  some  tangible  internal  management 
actions  that  place  an  additional  incentive  on  the  work 
quality.  A clear  explanation  of  the  essential  features  of 
the  incentive  structure  should  be  given  to  ail  who  work  in 
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Fiff.  7.  Penftlty  aosMment  structure  (case  history  B). 


any  manner  on  the  software,  keyed  to  the  contribution 
each  one  can  ake  to  affect  the  fee  that  can  be  realised 
by  his  company. 
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Fig.  8.  Typical  software  development  cost  experience  (case  his- 
tory B — multimodel  development). 
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Fig.  9.  Cost  distribution  by  activity  during  full  period  of  per- 
formance (case  histoiy  B — multimodel  development).  .\11  activities 
include  documentation  and  travel  costs.  Distribution  excludes 
OAM  costs. 


a. 


The  normalised  contract  cost  as  a function  of  the 
normalised  period  of  peifomumce  is  shown  in  Fig.  8 for 
case  history  B.  The  normalised  cost  distribution  by  de- 
velopment activity  over  the  software  life  cycle,  for  both 
IOC  and  FOC,  is  shown  in  Fig.  9.  In  estimating  the  cost 
of  large  software  systems,  we  treat  “operations  and  main- 
tenance” (O&M)  as  a level  of  effort  conunencing  im- 
mediately after  selloff  of  the  system  in  its  operational 
environment.  In  the  multimodel  software  development, 
O&M  of  the  first  model  overlaps  the  purely  development 
activities  of  the  second  model.  To  avoid  any  misinter- 
pretation of  the  development  costs,  the  O&M  amounted 
to  l.i  percent  of  the  total  contract  value.  Comparison  of 
case  history  .4  and  B in  sise  and  cost  (man-months), 
excluding  O&M  is  given  below: 


Number  of  Object 
Instructions 

Man-Months 

Rate 

(/) 

(\f) 

(I'M) 

Case  History  A 

102  400 

.570 

180 

Case  History  B 

IOC 

160  000* 

945 

170 

FOC 

215  000* 

630^ 

340 

* IOC  activities  included  design  and  pj^uction  of  a prototype 
software  package  of  an  additional  20  000-o^ect  instructions.  A 
slii^tly  modified  version  was  made  during  FOC. 

■>  Leaning  curve  phenomenon  has  been  verified  by  empirical 
data  and  controlled  tests  in  other  industries  [8|,  but  not  syste- 
matically for  the  software  industry. 

A Soflu'ore  Cost  Estimation  Algorithm 

TRW  has  developed  a cost  estimation  algorithm  based 
on  the  assumption  that  costs  vary  proportionally  with  the 
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number  of  instnirtions  For  each  identified  routine,  the 
prociHiure  combines  an  estimate  of  the  number  of  object 
instructions,  cateuorv.  relative  dt'gree  of  difficulty,  and 
historic  data  in  dollars  per  instruction  from  the  cost  data 
bns*'  to  i:i\e  a trial  estimate  of  the  total  cost.  The  design 
»£roup  estimates  the  first  three  stjftware  parameters.  The 

'tern  concept  group  provides  the  appropriate  cost  data 
iiase  ii,s<si  by  all  designers,  as  tvell  as  the  allocation  of 
re-sain’cs  to  each  phase  of  the  stiftware  development 
eycle,  the  schfslule  for  each  milestone  event,  and  the  laboi 
mix.  The  proee<lure  then  spreads  the  total  cost  over  the 
period  of  (terformance  according  to  these  input  param- 
eters, gives  the  man-months  per  month  by  labor  categorj’, 
the  cost  associated  with  each  development  activity,  and 
the  computer  usage  by  month  for  checkout  and  test 
.activitii's.  The  output  may  be  considered  a “trial  estimate” 
in  the  sen.se  that  the  propttsal  team  must  be  satisfied  with 
the  resulting  estimate  when  tested  against  any  conven- 
tional method  of  co.st  estimating. 

The  first  step  in  the  method  is  to  categorize  the  software 
routines  'hat  an>  being  considered  in  the  preliminary 
design.  The  software  categories  have  be.  i selected  based 
on  experience,  and  ate  those  functionally  different  kinds 
of  softAvare  entities  for  which  a significant  cost  per  in- 
struction (CPI>  is  expected.  Categories  that  have  stood 
the  test  of  usage  in  several  proposal  and  preliminary 
design  activities  are:  a)  control  routine,  which  controls 
execution  flow  and  is  nontime-critical.  C;  b)  input/output 
routine,  which  transfers  data  into  or  out  of  the  computer, 
/;  c'  pre-  or  po.stalgorithm  processor,  which  manipulates 
liara  for  .sub.se<|uent  proces.sing  or  output,  P-.  d)  algorithm. 
A.hich  perbimts  logical  or  mathematical  operations, 
.t;  e)  data  management  routine,  which  manages  data 
transfer  within  the  computer,  Z>:  and  f)  time-critical 
processor,  which  is  a highly  optimized  machine  dependent 
code.  T. 

The  next  step  is  size  and  complexity  estimates  by 
rriutine.  or  subprogram,  by  the  designer.  To  balance  cost 
and  risk,  the  designer  may  decide  to  use  software  elements 
that  are  available  to  him  fiom  a software  libraiy  and  need 
only  some  degiee  of  modification  or  adaptation.  At  the 
other  extreme,  a new  technique  may  be  required  and  be 
estimated  as  a high  technological  risk.  To  account  for 
the  degree  of  difficulty  of  a given  kind  of  routine,  the 
designer  estimates  a risk  or  complexity  factor.  This  is 
the  most  crucial  step  in  the  t'stimating  process,  for  it 
,*stablishes  the  cost  of  the  routine  with  all  dire<it  and  in- 
direct charges  amortized  against  it.  The  other  steps 
basically  determine  how  the  total  cost  will  be  spread  over 
the  development  cycle. 

Two  consultants  have  different  views  of  how  software 
parameters  should  be  estimated,  in  general.  Brandon  says 
a single  individual  should  establish  a complexity  rating 
scale  (A.B,C,D,E,F)  and  make  a “standard  estimate” 
for  each  job  based  on:  a)  complexity  rating  of  each  job; 


bl  machine  uused;  c)  language  used;  and  dl  estimated 
number  of  instructions.  Brandon  uses  equations  fitted  to 
hi.storical  data  to  get  standards,  and  then  measures 
performance  against  the  standard  [9], 

Lecht  believes  the  estimator  should  interview  the 
member  of  the  technical  staff  who  will  do  the  job,  and 
negotiate  personal  agreement  on  effort.  Historical  cost 
data  reinforce  the  estimator's  judgment  where  similar 
jobs  can  be  found.  The  estimate  is  based  on:  a)  .similarity 
with  previous  modules;  bl  person  doing  the  job;  cl 
machine  used:  d)  language  used;  and  el  estimated 
number  of  instructions.  Lecht  does  not  believe  meaningful 
performance  standards  can  be  set  for  softw’are  [lOj 
The  .simplest  technique  for  narrowing  down  the  many 
subjective  choices  early  in  the  estimation  process  is  to  ask 
is  the  routine  new  or  old.  and  is  it  easy,  medium,  or  hard? 
A complexity  rating  coefficient  can  be  applied  contin- 
uously from  1 to  20  as  a multiplier,  if  preferred.  The 
important  consideration  is  allowing  sufficient  degree  of 
freedom  to  accommodate  a teaming  experience  and  in- 
dividual differences  in  performance  in  the  effort  to  obtain 
a realistic  cost.  For  our  present  purposes  of  early  pre- 
liminary design,  we  will  allow  the  designer  four  choices 
to  account  for  six  levels  of  difficulty  for  each  routine.  They 
are: 


Easy 

Medium 

Hard 

Old 

OE 

OM 

OH 

New 

NE 

NM 

NH 

The  only  parameter  that  changes  as  a function  of  degree 
of  difficulty  fOE  through  XHl  is  cost  per  instruction. 
Override  control  is  available  to  the  designer.  Sample  data 
sets  will  be  given  in  the  section  on  Software  Cost  Data 
Base. 

The  next  step  is  to  identify  the  various  development  and 
test  phases  for  reducing  the  software  from  the  conceptual 
stage  to  delivery  of  the  operational  softwan*  to  the  ulti- 
mate user.  For  our  present  purposes,  the  seven  steps  given 
in  the  section  entitled  The  Software  Development  Cycle 
will  be  adopted,  but  they  could  be  any  other  steps  tailored 
to  the  needs  of  the  particular  application.  For  each  phase 
an  estimate  is  required  for  the  fraction  of  the  total  amount 
tc  be  allocated  to  it.  In  the  case  of  manpower  allocation 
for  distribution  of  1000  man-months  over  the  development 
cycle,  it  was  functionally  allocated  as  424  man-months  to 
design  and  analysis,  210  man-months  to  code  and  debug, 
and  366  man-months  to  check  out  and  test  (see  Fig.  o). 
This  distribution  of  42-21-37  percent  at  this  level  of 
detail  has  been  borne  out  in  practice  by  several  other 
researchers,  as  will  be  demonstrated  shortly. 

The  fourth  step  is  to  define  the  activities  in  each  de- 
velopment phase  by  means  of  an  activity  array  and  asso- 
ciattxi  cost  matrix.  The  activities  as  a function  of  develop- 
ment phase  is  a 2.5  X 7 matrix;  that  is,  25  activities  are 
identified  for  each  of  7 phnses.  In  turn,  subsets  of  the 
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activities  may  he  summed  to  “super  activity”  levels, 
and  the  makeup  may  vary  from  phase  to  pha.se.  A tA^pical 
activity  array  and  cost  matrix  will  be  presented  in  the 
section  entitled  Resource  Allocation. 

The  final  step  in  setting  up  the  initial  conditions  for  the 
cost  estimation  algorithm  is  to  provide  schedule  data 
based  on  the  customer’s  .statement  of  work,  or  other  man- 
agement considerations.  Scht>dule  data  are  input  as 
months  from  go-ahead  for  each  of  the  milestone  periods. 
Burden  rates  are  input  for  projected  overhead  rates, 
general  and  adniinstrative.  and  so  forth.  Labor  mix  is  de- 
fined and  unburdened  bid  rates  for  the  labor  grades 
desired  for  the  pha.se  are  defined.  Selective  cost  cuts  are 
under  management  control  by  the  override  capability. 
Other  direct  charges  are  input,  which  for  software  is 
typically  travel  as  a percentage  of  direct  labor  costs,  such 
as  percent  and  documentation  at  10  percent. 

Computer  usage  data  for  a machine  in  the  CDC  6.t00 
class  with  time-.shared  central  processor  unit  based  *on 
sampled  data  from  a programming  department  by  month 
have  been  [11] 


N umber 
of 

MTS 

Total  Number 
of  Processor 
Units  Used 

N umber  of 
Processor  Hours 
per  Man-Month 

Peripheral 
Hours  per 
Man-Month 

;)6 

5.5.6 

2.20 

7.6 

3.i 

31.0 

1 27 

4.4 

3;i 

30.0 

1.30 

4.5 

33 

46. .5 

2.02 

7.0 

-■Vllowing  for  slightly  less  power  in  a 370  l.'>.o  computer, 
for  instance,  a figure  of  3 h,  month  programmer  man- 
month  would  be  reasonable.  At  an  average  productivity 
of  1 object  instruction/h,  this  is  equivalent  to  l.ifi  in- 
structions man-month,  or  1.2  min  instruction.  The  data 
in  the  third  column  are  based  on  1.42  processor  units 
corresponding  to  Ih  of  computing  time.  .\  computer 
hours  matrix  in  terms  of  u.sage  rate  per  instruction  by 
phase  is  given  in  the  section  entitled  Resource  .\llocation 
basiHl  on  the  above  data. 

The  summary  output  from  the  cost  estimation  al- 
gorithm, SPREAD,  includes:  a)  cost  per  routine  based  on 
either  historic  burden  rates  or  proposed  burden  rates; 
b)  average  cost  per  instruction,  number  of  instructions, 
and  category;  cl  total  number  of  development  computer 
hours  per  routine;  d)  simple  graphic  display  of  schedule 
and  events;  e)  cost  breakdown  by  development  phase 
per  routine  in  total  dollars  and  percent;  f)  cost  break- 
down by  acti\’ity.  by  routine,  and  summed  over  all 
routines  in  the  sfiftware,  such  as  management,  review, 
documentation,  .spiniifications,  design,  coding,  and  test- 
ing; and  finally,  gi  manloading  and  cost  summary  by 
segment  showing  for  each  month  the  labor  breakdown 
for  senior  statT,  staff,  technical,  clerical;  also,  computer 
hours  by  month,  other  direct  charges,  cost  by  month,  and 
cumulative  cost. 


The  outputs  from  the  cost  estimation  algorithm  are 
considered  a “trial  set”  for  the  cost  e.stimation  group. 
The  trial  set  is  used  in  combination  with  all  other  sources 
of  data  to  ti'st  that  cost  position  against  the  project  ob- 
jectives. The  approved  trial  set,  which  contains  the  best 
judgmental  and  quantitative  measures  of  cost  per  soft- 
ware element  and  per  activity,  becomes  the  input  to  the 
official  pricing  computer  run.  Necessary  translation  of  the 
data  set  to  exactly  match  the  customer's  work  breakdown 
■Structure  (WBS^  or  other  appropriate  co.st  elements  is 
made  for  the  final  pricing  run.  The  official  cost  figures  are 
produced  by  cost  guidelines,  approved  rates,  and  pro- 
cedures that  have  been  established  by  the  in-house 
pricing  group  and  approved  by  the  government  auditor. 
The  results  of  the  software  cost  estimation  nlgorithm  are 
retained  as  cost  backup  data  and  for  possible  use  in  later 
cost  justification.  The  official  cost  figures  from  the  pricing 
computer  run  show  manloading  and  costs  collected  and 
aggregated  against  the  customer’s  WBS  ( see  Fig.  10) . 

This  cost  estimation  approach  has  the  following  ad- 
vantages. a)  The  amount  of  data  required  to  obtain  a 
cost  estimate  is  minimal,  b)  The  software  designer  im- 
mediately sees  the  cost  consequences  of  his  preliminary 
design,  c)  To  the  extent  that  the  routine  is  directly  trace- 
able to  a requirement,  the  requirement  is  directly  trace- 
able to  a total  cost,  d)  Comparisons  of  alternate  schedules 
on  resource  allocation  can  be  made  rapidly,  e)  A cost 
justification  is  produced  that  can  stand  the  test  of  govern- 
ment audit. 

SOFTWARE  COST  D.\TA  B.\SE 

Sensitivity  coi'fficients  or  exchange  ratios  u.sed  by  the 
cost  estimation  algorithm  reside  in  the  data  base.  The 
objective  in  cost  estimating  is  to  realize  greater  accuracy 
and  precision  while  anticipating  risk  and  its  impact  on 
profit.  The  greater  the  risk,  for  a given  confidence  level, 
the  greater  the  allowance  on  the  part  of  the  estimator  for 
this  uncertainty.  The  more  uncertain  the  estimate,  the 
less  competitive  is  the  cost  proposal.  Further,  an  estimate 
of  cost  backed  by  relevant  cost  experience  facts  gives 
confidence  to  negotiator,  management,  and  the  customer. 
Cost  justification  is  demanded  by  the  customer,  and  is 
usually  provided  in  the  form  of  lower  level  back-up  data. 

Valid  data  based  on  proven  expierience  are  required  for 
any  cost  estimation  process.  Producing  good  operational 
.software  is  dependent  on  the  many  activities  of  good 
people  properly  directed  and  motivated.  The  process  is 
difficult  to  measure  and  apply  wid  ly  beyond  the  tech- 
nology center  for  which  the  performance  measures  were 
derived.  There  is  no  universal  model,  .\pplying  “average 
productivity  rates”  without  knowledge  of  the  development 
steps  that  were  used  in  deriving  the  rates  is  wrong. 
.\verages  based  on  large  samples  are  useful  in  comparison 
of  a local  variable  against  a global  variable,  if  the  esti- 
mator is  reasonably  certain  he  is  comparing  truly  equiv- 
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Fig.  10.  Simplified  costing  sequence  for  TRW  software,  showing 
cost  estimating  algorithm  data  flow. 
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Fig.  11.  Cost  estimating  graph,  showing  industry-wide  averages. 


alent  measures.  Metzelaar  has  studied  the  problem  of 
industry-a-ide  productivity  rates  in  terms  of  program  size 
and  complexity,  as  shown  in  Fig.  11  Cl2]. 

The  activities  that  productively  occupy  the  analyst’s 
time,  along  with  the  plant  overhead  and  other  direct 
charges,  largely  determine  the  price  to  the  customer.  It  is 
in  the  company’s  best  interest  to  agree  on  measurable 
activities  that  should  be  available  in  the  software  cost 
data  base  for  cost  justification  and  estimation  of  new  work. 
Brief  definitions  of  activities  that  have  been  identified 
in  this  regard  are  given  below  £2]. 

I)  Learning  and  Orientation  (L):  Those  actions  neces- 
sary to  gain  understanding  of  the  hardware  and  software 
to  be  applied  in  the  project,  and  to  learn  the  operating  re- 
quirements, specifications,  and  constraints  to  be  satisfied 
by  the  software  package  being  produced. 


g)  Analysis  and  Design  (A):  All  actions  necessary  to 
generate  technical  solutions,  which  are  expected  to  satisfy 
requirements  and  specifications  within  the  imposed  con- 
straints. It  includes  determining  and  evaluating  technical 
approaches,  determining  and  evaluating  the  effects  of 
specific  requirements  and  constraints  on  potential  techr 
nical  solutions,  selecting  the  best  solution,  and  describiitg 
that  solution  so  it  can  be  coded.  This  activity  includes  the 
rough  documentation  that  is  normally  produced  in  gen- 
erating a technical  solution,  but  not  the  efibrts  of  finalising 
the  documentation  package. 

3)  Coding  (C):  The  translation  of  the  detailed  steps 
of  the  technical  solution  into  machine-readable  language; 
the  ordered  list,  in  computer  code  or  pseudocode,  of  the 
successive  coiaputer  operations  for  solving  a specific 
problem.  It  does  not  include  actual  input  to  the  computer 
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(except  for  time-share  inputs compiling,  nor  testing  the 
adequacy  or  validity  of  the  instructions. 

41  Test  attd  Checkout  {T'c  All  actions  necessary  to 
assure  that  the  instructions  coded  by  the  programmer 
cause  the  machine  to  do  what  the  programmer  intended 
it  to  do.  It  covers  preparing  the  test  data,  compiling  the 
program,  running  the  test  data,  reviewing  the  compiler 
and  or  machine  output,  identifying  errors,  correcting  the 
errors,  and  making  changes  to  the  program  that  improve 
its  reliability  and  efficiency.  It  does  not  include  verifying 
that  the  software  product  satisfies  the  requirements  and 
specifications  stated  by  the  customer  or  sponsor. 

J)  Verify: ''ucUity  iT');  Those  actions  necessary'  to 
f assure  that  the  software  product  satisfies  the  require- 

ments and  specifications  provided  by  the  customer  or 
•sponsor.  It  includes  preparation  of  qualifying  test  data, 
verification  runs,  evaluation  of  computer  run  results,  and 
adjustments  to  the  program  to  correc*  ny  deficiencies. 

System  Integration  and  Test  '/).  .ul  actions  necessary 
to  assure  that  the  subsystem  or  module  as  programmed 
will  interface  with  other  subsystems  or  modules  to  provide 
a satisfactory  system.  It  includes  preparing  data  for 
evaluating  the  joint  functioning  of  two  or  more  sub- 
systems or  modules,  making  the  evaluation  runs,  evaluat- 
' ing  the  joint  operation,  and  making  necessary  modifica- 
tions to  the  software  in  accordance  with  prevailing  con- 
figuration control  procedures. 

7)  Documentation  (D):  Those  actions  that  require 
technical  editing,  technical  typing,  art  work,  and  repro- 
duction of  deliverable  documents  to  the  customer  or 
sponsor. 

.\n  example  of  the  cost  per  instruction  for  the  sig- 
nificant categories  of  softwaue  previouslv  defined  versus 
degree  of  difficulty  is  given  in  I'ig.  12.  The  cost  figures, 
in  principle,  include  all  seven  development  steps  described 
in  the  section  on  software  development  and  test  manage- 
ment. .\ctivities  that  are  defined  immediatel}'  preceding 
are  inherc'ntly  included  in  the  cost  figures  as  are  all  other 
direct  charges  up  to  and  including  general  and  adminis- 
trative costs. 

In  late  1971,  TRW  compiled  in-house  data  on  a wide 
variety  of  software  development  characteristics,  which  in 
turn  were  independently  analyzed  by  Lulejian  and  .\ssoci- 
ates  under  contract  to  the  U.S.  Air  Force  [13],  [14]. 
Th*‘  variables  of  particular  interest  for  our  present  pur- 
posr-s  were  analyzed  by  Lulejian  and  .\ssociates  for  a 
package  of  SS  routines  developed  by  TRW.  A brief 
interpretation  of  tour  of  the  analyses  that  bear  on  the 
cost  of  developing  software  is  presented  next. 

Consider  I'ig.  13.  Th  s is  a plot  of  cost  versus  number 
of  instructions  for  the  .ss  routines.  Two  conclusions  can 
be  drawn,  either  by  direct  observation  or  by  regression 
analy.sis.  a)  There  is  a general  trend  in  the  data — larger 
routine's  cost  more,  b'  linear  fit  to  the  data  is  not  very 
good.  If  one  attempts  to  predict  costs  from  routine  size 
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Fig.  12.  Cost  per  object  instruction  versus  relative  degree  of 
difficulty. 


Fig.  13.  Relative  co.st  per  routine  versus  number  of  instructions. 

alone,  one  can  expect  errors  of  about  .50  percent  in  the 
predicted  cost.  The  prediction  is  better  than  nothing,  but 
not  much  better.  Fig.  14  shows  the  same  data  plotted 
another  way — cost  per  1000  instructions  versus  number  of 
instruct’ons.  If  the  cost  of  the  routine  can  be  estimated  by 
a fixed  cost  per  1000  instructions,  one  would  expect  this 
data  to  show  constant  cost.  It  appears  to  be  constant, 
with  a large  error,  especially  for  small  routines,  .\gain, 
one  concludes  that  cost  can  be  predicted  from  number  of 
instructions  provided  that  one  is  willing  to  accept  fairly 
large  errors. 

Cost  depends  on  a number  of  other  factors,  and  one 
might  hope  to  get  better  cost  estimates  by  including  dif- 
ficulty, programmer  experience,  etc.  A considerable 
number  of  fits  were  tried.  The  answers,  unfortunately, 
were  negative.  Including  other  variables  did  not  result  in 
an  equation  that  gives  any  significantly  better  fit  to  the 
data.  Fig.  1-5  shows  an  example  of  one  of  the  variables — 
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OF  ROUTiNe 
FER  1000 
INSTRUCTIONS 


NUMRER  OF  INSTRUCTIONS 


T.VBLE  I 

Multivariate  Regression  Summary 


GOODNESS 
OF  FIT 


Fig.  14.  l^)Utine  unit  versus  number  of  instructions. 


COST  • 0 2S  ♦ 0 00028  (NUMtER  OF  INSTRUCTIONS! 

COST  • 0.021  « 0 00018  (NUMBER  OF  INSTRUCTIONS! 

* 0.19  (DIFFICULTY! 

COST  - 0.006  ♦ 0.00029  (NUMBER  OF  INSTRUCTIONS! 

0 00011  'NUMBER  OF  LOGICAL  INSTRUCTIONS! 
0.00027  (NUMBER  OF  INRUT/OUTFUT  INSTRUCTIONS! 

* 0 OOBS  (NUMBER  OF  INTERFACES! 

<>0.138  (DIFFICULTY! 

^ 0.0034  (PERCENTILE  RATING! 

* 0.0080  I YEARS  OF  EXPERIENCE! 

0.0299  (YEARS  OF  SCF  EXPERIENCE! 

0.103  (SUPERVISORS  RATING! 


T.\BLE  11 

Computer  Program  Development  Breakdown 


IS.8!  (3.31  (8.81  (33! 


RELATIVE  COST  2.0 
OF  ROUTINE 
PER  1008 

INSTRUCTIONS  19 


• • • • ^ Nfc. 

.<•  .-J  • Nl* 

,1  1 1 I ^ « I I I,  >1 

IS  W 3*  40  M M 70  n M 

ExnRIINCt-SATINa  4ACTOII  OF  ROUTINC  FMMRMMfR 

Fi*.  l.i.  Routine  unit  cost  versus  experience  rating  factor  of  pro- 
grammer. 


experience  of  programmer.  .\  glance  at  the  curve  suggests 
that  a knowledge  of  this  variable  does  not  help  one  predict 
cost.  The  regression  analy.sis  results  support  this  sug- 
gestion. Table  I shows  the  results  of  a multivariate  linear 
regression  analysis,  .\dding  the  remaining  variables  gives 
a better  fit,  but  not  much  better. 

The  conclusion  is  that  there  are  no  simple  universal 
rules  for  costing  software  accurate!}'.  It  is  necessary  to 
understand  the  nature  of  the  individual  program  and  the 
individual  routines  within  the  program. 

RESOURCE  ALLOCATION 

Various  researchers  have  separately  discovered  by 
empirical  methods  a rule  of  thumb,  namely,  that  analysis 
and  design  account  for  40  percent,  coding  and  debugging 
account  for  JO  percent,  and  checkout  and  test  account  for 
40  percent  of  the  total  resource  (cost)  spent  for  software 
development,  the  40-20-40  rule.  If  the  cost  of  “modeling” 
of  physical  systems  is  to  be  borne  by  the  software  develop- 
ment group,  as  contrasted  to  modeling  analysis  from  raw 
data  by  technolog}'  centers  within  the  company  (such  as 
thermal,  attitude  control  and  pointing,  propulsion, 
electrical  power,  and  so  forth)  an  extra  .30  percent  typi- 


ANALYSIS 

AND 

DESIGN 

1 COOING 
AND 

AUDITING 

CHECKOUT 
AND  TEST 

SAGE 

39% 

14% 

47% 

NTDS 

30 

1 

20 

50 

GEMINI 

1 

38 

17 

47 

SATURN  V 

32 

24  I 

44 

cally  would  be  required  in  addition  to  the  software  develop- 
ment by  itself.  In  other  words,  the  deriving  of  equations  of 
motion  or  physical  behavior  of  other  physical  systems  from 
raw  aata  is  not  included  in  the  software  development 
resource  allocation.  Similarly,  the  cost  of  computing  time 
is  not  covered  in  the  resource  allocation.  Experience  has 
consistently  shown  that  computer  costs  add  an  additional 
‘20  to  25  percent  of  the  total  cost  of  the  project;  that  is,  a 
So  million,  year  software  development  contract  will  cost 
the  government  an  additional  SI  million  for  GFE  com- 
puting time  to  the  developer.  Boehm  discusses  his  views 
on  resource  allocation  in  Cl53;  his  findings  are  presented 
in  Table  II.  The  findings  of  an  in-house  survey  by 
Metzelaar  are  presented  in  Table  III.  Independently 
derived  resource  allocation  for  the  KXK)  man-month 
costing  example  was  given  in  Figs.  4 and  5. 

T e basic  resource  allocation  data  that  are  required  for 
application  of  the  cost  estimation  algorithm  are  shown  in 
explicit  form  for  purposes  of  illustration  only  in  this 
paragraph.  Table  IV  shows  a simplified  version  of  how 
"complexity”  might  be  handled  in  a preliminary  design. 
For  example,  if  we  are  designing  one  routine  in  a large 
system  that  we  estimate  to  have  KXK)  executable,  or 
object,  instruction,  (not  Fortran  IV  source  instructions), 
and  that  has  as  its  primary  function  setting  up  the  input 
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TABLE  III 

Average  Percentage  or  Analyst's  Time  by  Activity 


ACTIVITY 

INFORMATICS 

RAYTHEON 

TBV*^ 

AVG 

ANALYSIS 

20% 

20% 

20% 

20% 

DESIGN 

16 

20 

20 

18.7 

CODE 

16 

25 

24 

21.7 

TfST 

32 

25 

28 

283 

DOCUMENT 

16 

10 

8 

11  3 

100% 

100% 

100% 

100% 

MIX:  SCIENTIFIC 

160) 

(0) 

(100) 

(53.3) 

BUSINESS 

(401 

(100) 

(0) 

(46.7) 

* Substantial  variation  from  project  to  project. 


TABLE  IV 

Cost  Per  Instruction  as  a Function  or  Degree  of  Difficulty 
(Example  Only) 


DEGREE  OF 

CATEGORY 

C 

1 

F 

A 

D 

T 

DIFFICULTY 

OE 

621  00 

18.00 

17.00 

15.00 

24.00 

75.00 

OM 

27  00 

24.00 

23.00 

20.00 

3100 

75.00 

OH 

X.OO 

27  00 

26.00 

22.00 

35.00 

75.00 

NE 

33.00 

28.00 

28.00 

24  00 

37.00 

75.00 

NM 

40.00 

35.00 

34.00 

30.00 

46  00 

75.00 

NH 

46.00 

43.00 

42.00 

36.00 

56.00 

75.00 

REQUIREMCNTS  ANALYSIS 
PRELIMINARY  OCSIGN 
INTERFACE  DEFINITION 
DETAILED  DESIGN 
CODE  AND  DEBUG 
DEVELOPMENT  TESTING 

validation  testing 

AND  OPERATIONAL 
DEMONSTRATION 


IE 


20 


21 


X 


10 

PtOClNT 


Fig.  16.  Typical  al|pcation  of  resources  in  custom  software  de- 
velopment and  test. 


to  the  main  algorithm,  we  first  categorize  it  as  a “pre- 
processor,” or  category  P.  It  is  new  and  seems  to  be  of 
medium  difficulty  (compared  to  others  we  have  seen), 
therefore  it  is  further  categorized  as  new-medium,  or 
XM.  We  enter  on  the  worksheet  for  this  routine  its  name, 
say  PROG,  its  category  P/XM,  and  a number  of  ex- 
ecutable instructions  equal  to  1000.  The  consequence  of 
this  action  is  that  PROG,  at  134.00/instruction,  will  cost 
the  customer  (i.e..  directly  timeable  to  PROG)  S34  000. 
This  covers  the  cost  of  all  activities,  including  analysb, 
cV  sign,  documenting,  coding,  checkout,  and  test.  These 
activities  are  spread  into  the  development  phases  accord- 
ing to  the  typical  allocation  of  resources  shown  in  Fig. 
16.  Summing  the  first  four  phases  prior  to  code  and  debug 
shows  we  allocate  46  percent  of  our  total  dollar  there. 


coding  takes  20  percent,  and  the  two  major  phases  after 
coding  take  the  remaining  34  percent.  This  distribution 
has  been  tested  sigainst  at  least  two  other  computer  pro- 
gram development  cycles,  provides  a good  fit  to  present 
design  techniques,  and  will  be  adopted  as  the  nominal. 

The  resource  distribution  in  the  seven  phases  [steps 
l)-7)  previously  defined  correspond  to  the  seven  phases 
.4-6’]  will  vary  as  a function  of  “category,”  and  may  be 
interpreted  as  a perturbation  about  the  nominal.  In  all 
cases  H,  the  O&M  phase,  will  be  treated  differently 
since  it  is  not  characterized  by  the  same  objectives  or 
activities  as  the  development  phases.  The  percent  varia- 
tion of  total  resource  allocation  as  a function  of  category 
is  given  in  Table  V'. 

.\fter  classifying  the  software  package  (at  the  routine 


146 


\ 


«0LVKRTI>V:  OH'KLortVn  UROf-WiLt  som*A«K 


TABLE  V 

Ktxu'nri:  Alum'ation  a*  a Fi’nitios  or  Catxgort  for  Each 
Phase  i Example  Only) 


! 


! 

\ 

1 


PHASE 

CATEGORY  A ■ C O E 


G 


20 

2 

20 

14 

li 

4 

19 

21 

19 

3 

14 

23 

It 

1 

21 

23 

22 

4 

14 

It 

11 

12 

10 

24 

24 

21 

21 

17 

24 

21 


levplt  by  catPKpn*  and  deipw  of  difficulty,  and  defining 
the  dcvclopmont  phases  for  producing  the  software,  we 
define  the  “activities"  to  be  performed  during  each 
development  phase.  The  software  statement  of  work  is 
the  -source  authority  for  defining  activities  against  which 
casts  will  be  collected  at  an  appropriate  level  in  the  work 
■ breakdown  structure.  The  25  activities  (or  tasksi  that 
comprise  the  7 development  phases,  which  in  turn  lead 
to  preparation  of  the  deliverables  for  our  hypothetical 
project,  are  shown  in  Table  VI. 

The  assignment  of  resources  to  each  of  the  25  activities 
for  each  of  the  7 development  phases  is  shown  in  Table 
VTI.  Historical  data  from  the  cost  data  base  combined 
with  engineering  judgment  gained  from  work  on  previous 
similar  projects  are  used  in  making  the  assignment. 

The  remaining  step  is  to  assign  a properly  time-phased 
• computer  resource  to  be  used  for  the  development  and 
test  of  the  software.  The  rationale  for  such  a computer 
hours  matrix  was  given  earlier.  The  numerical  results  are 
. given  in  Table  VIII,  together  with  an  explanatory  note 
on  interpreting  the  data  array. 

An  example  in  the  application  of  the  entire  process 
would  be  useful.  Within  the  space  of  this  paper  it  is  not 
practical;  however,  preliminary  design  of  a large  command 
. . and  control  software  system  was  carried  out  in  late  1971 
that  provides  two  useful  insights  for  our  present  purposes. 
The  first  is  that  the  command  and  control  portion  of  the 
software  system  consisted  of  two  major  subsystems,  the 
first  containing  69  subprograms  which  were  considered 
. qtiasireal-time,  and  the  second  containing  29  subprograms 
which  were  considered  real-time.  In  fact,  some  of  the 
former  contained  real-time  subprograms  and  the  latter 
some  nonreal-time  subprograms.  Thus,  if  a casual  ob- 
server were  to  characterise  the  software  as  solely  real-time 
r (or  nonreal-time),  in  hopes  of  deriving  useful  cost  data 

i base  parameters,  he  would  be  certain  to  fail  in  precise 
interpretation  of  the  performance  data.  The  resource  allo- 
cation in  terms  of  burdened  man-months  and  computer 
hours  for  the  effort  is  given  in  Table  IX  at  the  module 
level,  where  routines  make  up  modules,  and  modules  make 
up  the  system.  If  this  system  had  been  carried  out  in  a 
complete  development  cycle,  a new  data  set  would  be 
entered  into  the  cost  data  base  and  varumces  between 
actual  and  predicted  resources  would  be  available  to  aid 
in  the  next  estimating  effort. 

I 


A second  insight  that  seems  useful  is  the  cost  estimating 
relationships  that  emerge  when  the  software  elements  are 
separated  into  meaningful  groupings  of,  in  this  case,  real- 
time, real-time  plus  its  supporting  environment  (pre- 
and  postprocessors  programmed  in  higher  order  language, 
for  example),  and  command  and  control  elements  that 
are  not  time-critical  (see  Fig.  17).  The  data  are  now 
highly  correlated,  and  the  differences  in  productivity 
rates  between  the  groupings  would  be  expected  to  range 
nearly  3 to  1.  The  observed  differences  should  not  be 
attributed  to  lack  of  correlation  between  dependent 
variables  that  are  seen  to  be  members  of  different  sets. 

TOPICS  FOR  FURTHER  WORK 

The  topics  that  were  originally  identified  for  discussion 
will  be  summarized,  together  with  key  issues  that  influence 
cost  estimating  of  large  software  systems  and  that  need 
further  work  and  study. 

I)  What  is  the  typical  code  production  rate  per  program- 
mer man-month"!  \ working  standard  may  be  inferred  from 
the  data  given  of  1 object  instruction /'man-hour,  which  is 
equivalent  to  156  instructions /man-month,  or  1870  in- 
structions, man-year,  or  6.4  man-months/ 1000  object  in- 
structions for  nontime-critical  software. 

If  a burdened  man-month  varies  between  14300  and 
$4900  in  1972  dollars  for  a typical  programming  depart- 
ment’s labor  mix,  on  the  average  this  working  standard 
translates  to  $27.50-131.50  as  the  cost  per  instruction. 
Both  case  history  A and  B correlate  strongly  with  these 
standards.  They  are  characterized  as  large  software  jobs 
whose  development  is  firmly  structured  and  controlled. 
But  what  about  the  R&D  work,  not  .well  structured  and 
controlled,  and  which  therefore  does  not  lend  itself  to 
estimation  by  extrapolation  from  a historical  cost  data 
base? 

The  underlying  question  is  whether  we  have  identified 
all  the  right  parameters  to  capture  and  quantify  in  our 
cost  data  base.  Do  we  understand  the  causal  relationships, 
and  can  they  be  isolated  from  their  effects  or  s}rmptom8? 
For  example,  is  the  SAGE  data  point  where  it  is  in  Fig.  11 
because  it  is  “extremely  complex”  or  because  the  require- 
ments were  poorly  defined? 

S)  How  does  code  production  rale  vary  with  problem 
complexity?  With  more  sophisticated  physical  systems 
being  brought  under  software  conunand  and  control,  a 
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TABLE  VI 

AcTtv-mEs  AS  .A  Function  op  Soptwxre  DtvxujPMeNT  Prase 


[ 


^''^^£VlLQW*CNT 

PHASE  A 

PHASE! 

phase  C 

PHASED 

1 PHASES 

phase  p 

PHASE  c 

PHASE  H 

activity 

PENEQNMANCE 
AMO  DESIGN 
AEOUiAEMlNTS 

IMPLEMENTATION 
CONCEPT  ANO 

1 TEST  PLAN  1 

INTCIirAClM6 

1 SftCIflMriOR 

OCTAILEO  OESIGN 
SMCIPICATION 

COOING  ANO 

1 AUDITING 

STHEM 

TESTING 

CERTIPICATION 

AMO 

acceptance 

operations 

ANO 

MAINTENANCE 

1 

, pnocnam 

MANAGEMENT 

PROGRAM 

MANAGEMENT 

1 PROGRAM 

1 management 

PROGRAM 

management 

program 

MANAGEMENT 

PROGRAM 

MANAGEMENT 

2 

PNQGBAM 

CONTAOL 

PROGRAM 

CONTROL 

PROGRAM 

CONTROL 

PROGRAM 

, CONTROL 

PROGRAM 

CONTROL 

PROGRAM 

CONTROL 

ncvi(ws 

3 

intenpace  I 

DESIGN  1 

REVIEW  UONl  ' 

critical 

OESIGN 

REVIEW  ICOR) 

SYSTEM  TEST 
REVIEW  ISTR) 

acceptance 

TEST  REVIEW  tATR) 

operational 

CRITIQUU 

DOCUMENTS 

^ 4 

DOCUMENT 
AND  EDIT 

DOCUMENT 

1 ANO  EDIT 

DOCUMENT 
AND  EDIT 

DOCUMENT 

AMO  EDIT 

1 DOCUMENT 
ANO  EDIT 

DOCUMENT 

ANO  EDIT 

DOCUMENT 
ANO  EDIT 

s 

NEPNOOUCTION 

wMooueno* 

REPROOUCTION 

REPROOUCTION 

REPROOUCTION 

RIPNOOUCnOH 

REPRODUCTION 

E 

NEQUiNIMfNTS 

OEPINITION 

KmH' 

PRODUCT  CONPIG 
OETAIiCO  TECH 
OCSCRiPTtON 

M 

RSQUIRIMENTR 

CERTIPICATION 

SOPTWARE 
PROKEM 
REPORTS  (SPR) 

NEQUlMMfNTS 

1 7 

NEOUINEMINTI 

AllOCATION 

r 

perponmahce 

ANO  DESIGN 

command  I 

OEPlNinON  l/P  ^ 

iPARTin  ■ " " 
nUTHQUr  LUTINGS) 

TRAINING 

DOCUMENTATION 

iMTN  LISriNOS) 

SftCIElCATlONt 

t 

h 

AEilUlAEMENTS 
(PANT  U 

telemetry 
OEPiNinoH  I/p : 

tHURPACi 

tffClPICATtONS 

iMATf 

9 

I 1 

1 

OPERATIONAL 
ENVIRONMENT  i/P 

D 

TNAOI 

STUDIES 

1 TNAOE 

STUDIES 

II 

trade 

STUOlU 

D 

FUNCTIONAL 

OEPINITION 

DATA 

OEPINITIONS 

algorithm 

DESIGN 

algorithm 

UPDATE 

1 PROGRAM 

UPDATE 

OCSlCN 

fl 

PROGRAM 

DESIGN 

! 

B 

STANOANOi 

AMO 

CONVENTION! 

DATA 

lASE 

OEPINITION 

DATA  SAtf 

OESiGN 

DATA  RASE 
UPDATE 

DATA  RASE 
UPDATE 

B 

tOPTWANI 

OVCNVIEW 

1 (PNELIMNANVI 

COOtttO 

D 

PROTOTYPE 

COOING 

operational 

COOING 

B 

, PNOOUCT  ANO 
CONPIGUNATION 

1 CONTNOL 

PRODUCT  ANO 

CONPIQURATION 

CONTROL 

PRODUCT  ANO 

CQNPIGURATION 

CONTROL 

PRODUCT  ANO 

CONPIGURATION 

CONTROL 

PRODUCT  ANO 

CONPIGURATION 

CONTROL 

PRODUCT  ANO 

CONPIGURATION 

CONTROL 

PRODUCT  ANO 
CONPIGURATION 

control 

B 

DATA  USE 
CONTNOL 

OATASASf 

CONTROL 

DATA  RASE 
CONTROL 

OATAIASS 
CONTROL  j 

OATARASC 

CONTROL 

DATA  RASE 
CONTROL 

OATASASE 

CONTROL 

TfSTINC 

CONEIGUNATION  | 
CONTNOV.  AMO  ; 

B 

TEST 

NEQUlNEMENTt 

TUT  1 

PLAN! 

INTERPACES 

TEST 

PROCEOURU 

DEVELOPMENT 

TtniNG 

PLANNING 

SYSTEM  TEST 

PLANNING 

ACCEPTANCE 
ANO  TEST 
PUNMNG 

INTEGRATION 

TESTING 

D 

1 

1 

DEVELOPMENT 

TEST 

SOPTWARE 

lYSnMTEST 

acceptance 

DEMONSTRATION 

rR  CLOSURE 

ONA 

i 

2f 

1 

1 

1 

! 

B 

acceptance 

TET 

NEQUINEMEMn 

QUALITY  ANO 
NELlAHLirY 

AttUNANCE  PLANS 

QAANO  RA 
MONITORING 

QAANORA 

MONITORING 

QANORA  1 

MONITORING  | 

QANORA 

MONITORING 

Q ANO  RA 
MONITORING 

Q ANO  RA 
MONITORING 

1 

B 

rttT  sumiiiT  1 

TEST  SUPPORT 

TEST  SUPPORT 

TEST  SUPPORT 

23 

1 

j 

OAENATIOMO 

B 

OPINATIONAL  ' 
CONCEPT 

OPERATIONAI 

rmcLiN! 

OPERATIONAL 
CONCEPT  UPDATE 

USERS  MANUAL 
IPRiLIMINARV) 

operational 
timilim  update 

USERS  manual 
UPOATE 

INTEGRATION 

SUPPORT 

n 

TNAtWMB 

PLAN 

TRAINING 

PUN  UPDATE 

- - - - J 

TRAINING 

TRAIHINO 

TRAINING  AND 
RSMEARUL 

TRAlNiNG  ANO 

rehearsal 

r • 


conrsponding  incrnaso  in  t^hnological  risk  and  com- 
plexity causes  an  increase  in  a responsible  cost  estimate. 
Reliable  estimates  place  this  increase  in  cost  right  now 
from  to  o..)  times  the  average  of  our  historical  cost  data 
base.  The  leading  causes  of  these  predicted  increases  need 
serious  attention  and  .study,  and  effective  management 
methods  to  detect  and  deal  with  them.  For  example,  we 


have  already  observed  in  two  cases  that  the  cost  of  pro- 
ducing real-time  (or  timc-critical)  software  is  three  times 
more  costl.v  than  nonreal-time  software  (see  Figs.  12  and 
17).  Should  the  development  dollar  be  spent  on  highly 
optimized  machini'-dependent  code  by  the  applications 
programmer  or  on  more  efficient  compilers  and  assemblers? 

Cost  tradeoffs  are  needed  between  the  crucial  param- 
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TABLE  VH 

Cost  Matrix  Data  Showing  Alux;ation  or  Resocrcbs  as  a 
Fi-NcnoN  or  Acnvrrr  bt  Phase  (Categort  P) 


PHAU 

ACTIVITY  A»iaiia!l 


(SI 

ii*i 

131 

(141 

(23) 

liii 

(12) 

(0) 

1 

10 

0 

• 

0 

7 

9 

10 

3 

2 

1 

3 

3 

3 

3 

3 

3 

8^ 

3 

• 

4 

0 

3 

8 

8_ 

4 

13 

• 

$ 

0 

5 

5 

4 

2 

S 

5 

2 

2 

3 

3 

2 

2 

1 

'• 

22 

• 

7 

12 

3 

7 

8 

8 

7 

10 

• 

7 

2 

8 

• 

7 

9 

• 

0 

_ 

10 

17 

10 

10 

S 

11 

2 

10 

10 

f 

7 
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T.\BLE  VIII 

CoMPVTBR  Hours  Matrix  SROwaNG  Computer  Usage  Allocation 
AS  A Function  or  Prase 


TABLE  IX 

Resource  Allocation  roR  Large  ComiAND  and  Control 
Software  System  (Preliminart  Design) 
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eters  that  define  the  problem  complexity  and  the  cost 
benefits.  A marginal  utility  theory  is  needed  for  the 
economics  of  software  development.  Parameters  that  are 
crucial  in  defining  problem  complexity  include  degree 
and  time  phasing  of  simulation,  extent  of  prototype  code, 
parallel  development  of  different  formulations  of  key 
algorithms,  new  computer  hardware/software,  assembly 
language  coding,  multisite  operations  and  interchange- 
ability,  growth  requirements,  critical  timing  and  through- 
put, hardware  software  interfaces,  fault-tolerant  com- 
puting, core  occupancy  constraints,  reliability,  and 
safety  [16K18]- 

5)  Hove  doe*  this  rate  vary  a*  a function  of  computer 
availability'f  Accessibility*  Configuration^  The  President’s 
Blue  Ribbon  Defense  Panel  reported  in  June  1970  in  its 
staff  report  on  automatic  data  processing,  that  because 

> of  the  government's  method  of  selectionand  procurement  of 
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ExampU:  The  column  sum  of  catofory  C,  control  routine,  is 
217  X lO-*  h instruction,  or  1.4  min/instruction.  The  total  resource 
required  is  then  distributed  over  phases  D-O.  as  shown  above,  e.g., 
0.42  min/instruction  for  phase  B,  coding  and  auditing. 

computer  s^'stems,  multiple  computers  in  the  same  geo- 
graphical area  can  result  in  costs  that  are  as  much  as 
five  times  larger  than  would  be  necessary  if  a few  large 
computers  were  used  in  a shared  operating  mode.  The 
underlying  problem  here  is  that  of  data  privacy  and 
protection  (security)  for  enabling  the  use  of  r«note 
terminals  by  government  agencies  for  both  classified  and 
unclassified  werk.  An  independent  evaluation  by  E.  C. 
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FIr.  17.  Coot  estimating  relationships,  showing  wide  variation 
arithin  single  rommand  and  rontrol  .software  system. 


Nolson  of  TRW  is  that  a dedicated  computer  installation 
costs  three  to  five  times  as  much  to  operate  as  if  it  were  a 
time-shared  installation  with  proper  remote  terminals. 
If  private  industry  can  enjoy  these  benefits  of  cost  ef- 
fectiveness by  use  of  remote  terminals  ( three  to  five  times 
less  expense),  it  would  seem  that  some,  if  not  a large 
part,  of  government  installations  could  enjoy  the  same 
benefits  if  the  problems  of  data  privacy  and  protection 
were  .solved  to  every  b<Kly’.s  satisfaction. 

management  goal  is  to  do  the  difficult  parts  of  the 
software  twice — once  to  produce  prototype  code  (>arly  in 
the  development  cycle  using  perhaps  S-10  percent  of  the 
resources  and  an  improved  version  with  the  technological 
risk  eliminated  for  the  operational  version.  One  pitfall  is 
that  the  risk  is  highest  in  procurements  where  a new 
computer  and  its  operating  system  are  being  cfeveloped 
concurrently  with  advanced  applications  software,  so  just 
when  computer  time  is  needed  the  most,  it  is  available  the 
least.  More  planning  to  reduce  the  computer  .selection  and 
procurement  time  is  needed. 

lurthermore,  in  some  operational  systems,  the  com- 
puter configuration  available  to  the  user  is  significantly 
different  from  the  computer  configuration  available  to  the 
computer  program  associate  contractor.  Such  a working 
environment  requires  an  integrating  contractor  to  main- 
tain system-level  configuration  control.  This  may  add  10 
percent  to  the  total  cost  and  is  viewed  as  an  insurance 
policy.  To  eliminate  such  occurrences,  standards  for 
computer  configurations  are  needed,  and  configuration 
control  is  needed  by  the  associate  contractor  and,  e(]uaily 
importantly,  by  the  user. 

4)  What  are  the  pieces  of  information  required  to  make  a 


realistic  prediction  of  software  development  cost?  W'e  have 
identified  the  activities  that  occupy  the  analj'st’s  time  in  a 
25  X 7 matrix  (Table  VI)  and  elsewhere.  However  it  is 
one  thing  to  identify  significant  activities  for  allocation 
of  resources,  and  it  is  another  thing  to  hold  the  right  set  of 
individuals  accountable  for  the  expediture  of  those  re- 
sources for  those  things  over  the  life  cycle  of  the  software 
development,  which  spans  12-24  months  or  longer.  The 
purpose  of  a cost  estimating  system  is  to  reduce  the 
variance  between  estimates  of  what  a software  develop- 
ment should  cost  and  what  it  actually  docs  cost.  The  cost 
history  data  file  is  a key  element  in  an  improved  cost 
estimating  system.  Further  work  is  needed  in  capturing 
and  maintaining  historical  estimates,  costs,  and  quanti- 
tative facts  to  provide  more  effective  data  for  estimating 
future  costs  and  sizes.  Further  work  is  needed  in  main- 
taining cost  control  over  the  actual  performance  period 
considering  the  following. 

a)  Recording  and  displaying  of  both  individual  and 
aggregate  programmer  activities,  such  as  previously  de- 
fined by  L,A,C,T,V.I,  and  D,  by  easy-to-use  automated 
project  control  and  prediction  tools. 

b)  Fast  computation  of  project  cost-  and  time-to- 
complete  predictions  by  current  and  look-ahead  activity 
statistics,  including  uncompensated  ovettime,  as  is  now 
feasible  thn^ugh  automatiHl  cost  control  tools. 

ci  Provision  for  storing,  n'trieving,  and  tracing  of 
projwt  stati'ment  of  work  deliverables  to  their  cost 
estimate  and  cost  estimate  to  actual  cost,  by  activity, 
over  the  us«>ful  life  of  the  software  through  cost  control 
tools. 

5)  How  does  code  production  rate  vary  as  a function  of 
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praijrammer  qualih)?  Quality  requirements  on  the  final 
product?  The  point  to  be  made  here  is  that  we  have  seen 
convincinft  evidence  of  the  validity  of  the  40-20-40  rule. 
For  a •$.")  million  project.  S2  million  will  be  spent  on  check- 
out and  test  to  the  extent  the  rule  applies.  .\  manattement 
objective  is  to  check  out  ever\'  loiric  path  in  the  software 
to  deliver  an  error-free  program.  This  is  a high-quality 
final  product  insofar  as  being  error-free  is  a measure  of 
quality,  .\chieving  this  objective  may  add  significantly 
to  the  development  cost  but  reduce  even  more  significantly 
the  operations  and  maintenance  cost.  With  improved  test 
tools  and  flow  measurement  tools,  considerable  improve- 
ment in  efficiency  of  more  branches  checked  per  computer 
hour  becomes  realizable.  From  the  analysis  given  for  one 
software  system,  the  quality  of  the  programmer  in  vears 
experience  and  supervisor’s  rating  seemed  to  make  no 
difference  in  the  cost  in  terms  of  the  goodness  of  fit,  the 
adjusted  index  of  determination  fr*)  given  in  Table  I 
and  Fig.  l'>.  As  R.  J.  Hatter  says,  some  of  the  signs  exhibit 
counter-intuitive  behavior. 

Technical  performance  standards  and  criteria  are 
needed  in  the  software  development  process.  .\  wrong 
standard  is  better  than  no  standard  because  at  least  it 
gives  a sense  of  direction  to  the  effort.  technical  per- 
formance measurement  system  is  needed  for  applying  to 
the  software  development  process,  and  which  is  under- 
stood and  applied  at  all  management  and  performer  levels 
over  the  life  cycle.  Since  software  development  is  more 
process  oriented  than  product  oriented,  such  standards 
become  measures  of  group  productivity  if  not  individual 
productivity.  In  some  organizations  it  may  be  interpreted 
to  be  again.st  company  policy  to  measure  and  report  on  a 
productivity  index  of  an  individual  member  of  the  pro- 
fessional staff,  such  as  code  production  rate,  or  error  rates 
(as  measured,  for  instance,  by  software  problem  reports). 
Producing  quality  custom  software  is  a profit-motivated 
. business,  and  more  businesslike  procedures  are  being 
applied  as  the  product  lines  mature  [193. 

6)  Hone  does  cost  vary  with  completeness  of  problem 
i . formulation?  Using  the  40-20-40  rule  again,  but  t^  time 
dealing  with  the  "analysis  and  design"  40  percent  (which 
for  our  example  of  Fig.  16  was  46  percent),  we  have 
evidence  that  analysis  and  design  takes  the  lion’s  share  of 
the  software  development  dollar.  A management  goal  that 
has  proven  effective  in  the  past  is  to  produce  superior 
documentation  which  is  reviewed  with  a knowledgeable 
technical  staff  in  the  customer’s  complex.  Thorough  and 
continuous  involvement  of  the  customer  in  the  develop- 
ment process  has  been  a reality  of  several  large  software 
developments.  Nothing  takes  the  place  of  competence 
and  communication  when  it  comes  to  understanding  the 
customer’s  or  sponsor's  requirements.  Translating  total 
system  requirements,  which  may  not  be  well  understood 
even  by  the  customer,  to  the  software  system  requirements 
r is  a crucial  first  step.  In  the  requirements  definition  and 
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preliminary'  design  phase,  there  is  typically  a lack  of 
data  but  lots  of  leverage  f influence  on  the  ultimate  design 
outcome),  in  contrast  to  the  operational  phase  where 
there  are  lots  of  data  but  little,  if  any,  leverage.  This  is  a 
significant  area  for  cost-  and  performance-effective  im- 
provement. 

7)  What  is  the  role  of  “desiqn-to-cost”  in  the  development 
of  larqe-scale  software?  The  DOD  joint  logistics  com- 
manders have  recently  entered  into  an  agreement  con- 
cerning the  acqu-sition  and  ownership  of  major  weapon 
systems  based  on  the  concept  of  “design-to-cost”  [20]. 

The  term  “design-to-cost”  is  defined  as  a process  utilizing 
unit  cost  goals  as  thresholds  for  managers  and  as  design 
parameters  for  engineers.  Cost  parameters  fgoab)  are  to 
be  established  that  translate  into  “design  to”  require- 
ments. In  the  past,  the  order  of  priorities  for  major 
weapon  system  acquisition  and  ownership  has  been: 

a)  performance,  b)  availability,  and  c)  cost.  Today, 
under  the  design-to-cost  doctrine,  the  order  is:  a)  cost, 

b)  performance,  and  c)  availability.  The  applicability 
of  the  design-to-cost  concept  to  hardware  is  now  beginning 
to  emerge  and  is  being  reduced  to  practice.  The  relation- 
ship of  the  design-to-cost  concept  to  software  is  not  now 
understood. 

In  the  197.5-1980  era,  this  reordering  of  priorities  for 
major  weapon  systems,  of  which  software  is  generally  an 
integral  part,  is  expected  to  have  a sizeable  influence  on 
both  hardware  and  software  life  cycle  acquisition  and 
ownership,  particularly  in  the  sense  that  the  unit  cost 
requirement  will  become  a driving  parameter  in  the  design 
of  large-scale  software.  Sj'stem  development  will  be 
continuously  evaluated  against  the  unit  cost  requirements 
with  the  same  rigor  as  now  applied  to  technical  require- 
ments. Practical  tradeoffs  must  be  made  between  system 
capability,  cost,  and  schedule.  Traceability  of  estimates 
and  costing  factors,  including  those  for  economic  escala- 
tion, will  have  to  be  maintained. 
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hnprovtmttus  in  pro^ntinnnnv  tet  hnuluny  have  parulMed’  im- 
provements in  computing  \yMem  architecture  and  materials. 
Alunit  with  increasiufi  knowledge  of  the  system  and  proyram 
development  processes,  there  has  been  some  notable  research 
into  proyrumminv  project  measurement,  estimation,  and  plan- 
ning. Discussed  is  a method  ot  proyrumminy  project  productiv- 
ity estimation.  .4 Iso  presented  are  preliminary  results  ot  re- 
search into  methods  of  measuriny  and  estimatiny  proyramminy 
project  duration,  staff  size,  and  computer  cost. 


A method  of  programming  measurement  and  estimation 

by  a E.  Walston  and  C.  P.  Falix 

New  materials  technologies  and  architectures  are  significantly 
affecting  computing  system  hardware.  While  the  cost  per  bit  of 
storage  and  the  execution  cost  per  instruction  have  both  been 
decreasing,  the  same  trend  has  not  been  true  for  software.  Since 
software  development  has  continued  to  be  a people-oriented  ac- 
tivity. a higher  percentage  of  the  cost  to  acquire  a computing 
system  is  accruing  to  software  development.' 

New  management  and  programsning  techniques  have  been  de- 
veloped to  improve  programming  efficiency.  Among  the  im- 
proved progranuning  technologies  are  the  following: 

• Chief  proyrammer  team,  a programming  organization  built 
around  a chief  programmer,  a backup  chief  programmer,  and 
a librarian  that  effectively  produces  code  in  a disciplined  and 
open  environment.' 

• Hierarchy  plus  Input-Process-Output  (HIPOK  a graphic  de- 
sign and  documentation  method  that  is  used  to  describe  pro- 
gram fimetions  from  the  topmost  level  to  great  detail." 

• Development  support  library,  a tool  that  provides  current 
infonnatioa  about  praiect  programs,  data,  and  status." 

• Structured  proyramming.  a programming  method  based  on 
the  matheriBticaJ  Structure  Theorem*  that  enables  program- 
mers to  understand  and  enhance  programs  that  have  been 
wrinen  by  others.*  as  well  as  one's  own  programs. 

• Design  and  code  inspections,  a review  of  program  design, 
code,  and  documentation  to  detect  errors  prior  to  program 
execution.* 

• rop-doN’ndrvr/opmrnr.  an  ordering  of  program  dc</clopment 
and  testing  that  begins  at  the  topmost  ftinctional  level  and 
proceeds  decrementally  to  the  lowest  ftinctional  level. 
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This  paper  discusses  research  into  programming  measurements, 
with  emphasis  on  one  phase  of  that  research;  a search  for  a 
method  of  estimating  programming  productivity.  The  method  we 
present  is  aimed  at  measuring  the  rate  of  production  of  lines  of 
code  by  projects,  as  influenced  by  a number  of  project  condi- 
tions and  requirements.  We  do  not.  however,  measure  the  per- 
formance of  individual  programming  project  members.  As  we 
continue  our  research,  we  are  continuing  to  learn  more  about  the 
attributes  of  the  programming  process,  about  programming  it- 
self. and  about  better  ways  of  analyzing  the  data. 

Before  starting  the  progianuning  measurements  research  re- 
ported here  we  analyzed  the  literature  on  programming  measure- 
ments and  productivity  that  includes  the  work  of  Aron.*  Wein- 
wurm  and  Zagonki.*  and  Nelson.'*  We  also  wanted  to  isolate 
the  effects  of  improved  programming  technologies  from  the  ef- 
fects of  monetary  inflation,  variations  in  computing  cost,  and 
ambiguities  in  the  definition  of  computing  quantities  (such  as 
that  of  the  size  of  the  delivered  program  product) . 

The  software  measurements  project  began  in  1972  when  we  de- 
cided to  assess  the  effects  of  structured  programming  on  the 
software  development  process.  To  do  that,  a rigorous  program 
was  established  to  measure  the  then  current  software  method- 
ologies so  that  changes  that  result  from  the  introduction  of  new 
methodologies  could  be  measured.  The  initial  phase  of  the  mea- 
surements program  was  to  identify  variables  to  be  measured,  to 
design  a questionnaire  (called  the  Programming  Project  Sum- 
mary questiotmaire) . and  to  develop  a system  for  processing  the 
data  to  be  collected.  The  first  report  entered  the  system  on 
January  23.  1973,  and  data  have  continued  to  be  received  and 
entered  into  the  system  ftom  that  time  on.  Since  the  establish- 
ment of  the  Prograituning  Project  Summary,  two  new  reporting 
formats  have  been  designed:  the  Software  Development  Report 
and  the  Software  Service  Report. 

We  understood  at  the  outset  that  the  established  objectives 
might  change  as  the  project  matured  and  as  data  were  accumu- 
lated. Current  objectives  of  the  project  discussed  in  this  paper 
arc  the  following; 

• Provide  dau  for  the  evaluadon  of  improved  programming 
technologies. 

• Provide  support  for  proposals  and  contract  performance. 

• Gather  and  preserve  historical  records  of  the  software  de- 
velopment work  performed. 

• Provide  progranuning  data  to  management. 

• Foster  a coitunon  progranuning  terminology. 
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Report  Same 

Sature  of  the  Report 

Progniinming  Project 
Summarv 

► 

[ 

Detailed  repon  on  the  wftware  development 
environment,  the  product  'including  errors i. 
resources,  and  schedule. 

Software 

Oevelopmeiit  Report 

i 

Detailed  repon  on  the  software  development 
envirooment,  the  product  i Including  changes  and 
etrori).  resources  and  schedule: 

Monthly  Software 
Devalopmem  Report 

One  report  on  product,  resource,  and  schedule 
status.  Changes  in  software  developiDeiit 
environiiiem  are  noted. 

Software  Service 

Report 

Report  00  prpfcct  size  and  on  the  software 
service  environment 

Quarterly  Software 
Service  Repon 

Detailed  data  on  product  being  serviced, 
resource  status,  and  changes  in  software  service 
environinenL 

Key  to  a software  tneasuienient  prograin  are  the  analysis  of 
measuremenu  and  feedback  of  results  to  the  suppliers  of  the 
raw  dau  and  to  management.  This  paper  discusses  the  data  re- 
porting and  analysis  in  the  software  measurements  program  of 
the  IBM  Federal  Systems  Division.  Described  first  are  the  mea- 
surements data  base,  the  services  that  are  available  to  users  of 
the  data,  and  descriptive  statistics  from  the  data  base.  The  next 
section  covers  the  analysis  of  progranuning  productivity  and 
describes  a productivity  estimation  technique.  Following  this, 
the  results  of  other  analyses  that  have  been  performed  with  re- 
spect to  factors  such  as  documenudon.  staffing,  and  computer 
costs  are  presented.  The  last  section  briefly  describes  analysis 
efforts  that  are  being  contemplated  for  future  research.  In  the 
measurements  project  discussed  here,  a sufficiently  large  quanti- 
ty of  data  is  being  collected  so  that  a programming  project  mea- 
surement system  is  used  to  provide  the  necessary  flexibility  and 
capability  of  storage  control  and  analysis.  The  Programming 
Project  Measurement  System  is  briefly  described  in  the  Appen- 
dix. 
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Data  contained  in  questionnaires  submitted  by  line  projects  at 
prescribed  reporting  periods  are  stored  in  a computer  data  base 
where  they  are  accessible  for  answering  queries,  preparing  re- 
ports. or  for  analytical  studies. 

The  basic  measurements  data  base  is  structured  so  that  it  con- 
tains all  the  reports  received,  as  described  in  Table  I.  Each  Pro- 

56  WaUTON  and  FBLIX  IBM  SYST  J 


156 


en 


i 


I; 

li 


[ 


1 


l» 


gramming  Project  Summary  milestone  report  is  given  a unique 
number  when  received  and  is  assigned  a separate  record  in  the 
hie.  Project  milestones  are  the  following:  start  of  work:  prelimi- 
nary design  review:  midway  through  software  development: 
acceptance  test  completion:  and  every  three  months  during  the 
maintenance  or  service  phase.  Each  initial  and  final  Software 
Development  and  Software  Service  report  is  also  tiled  as  a sepa- 
rate record. 

The  monthly  Software  Development  Reports  plus  any  changes  to 
the  initial  submission  constitute  separate  records,  as  do  the 
quarterly  Software  Service  Reports  and  any  changes  to  the  ini- 
tial Software  Service  Report.  Each  record  is  structured  into 
heids.  or  variables,  that  correspond  to  the  question  response 
fields  in  the  questionnaires. 

Sixty  completed  software  development  projects  are  now  in  the 
data  base.  These  completed  projects,  which  represent  a wide 
variety  of  programming  technology,  are  summarized  in  Table  2. 

Their  delivered  source  lines  of  code  range  from  4000  to  467 000, 
and  their  effort  ranges  from  12  to  1 1 758  man  months.  These 
programs  include  real-dme  process  control,  interactive,  report 
generators,  data-base  control,  and  message  switching  programs. 

Some  of  the  programs  have  severe  doling  or  storage  constraints 
and  other  programs  have  both  types  of  constraints.  Twenty- 
eight  diflbrent  high-level  languages  and  66  different  computers 
are  represented  in  the  listing  in  Table  2. 

With  the  previously  descnbed  data  base  and  with  the  capabili-  •erriem 
des  provided  by  the  Programming  Project  Measurement  Sys- 
tem. a wide  variety  of  services  can  be  provided  to  line  project 
personnel.  Queries  can  be  answered,  analyses  performed,  and 
programming  project  productivity  estimation  provided  for  new 
or  on-going  projects.  Examples  of  services  are  discussed  through- 
out the  remainder  of  this  paper. 

Queries  that  can  be  answered  by  direct  retrieval  of  data  from  the 
data  base  are  of  two  general  types.  First  is  a request  for  data 
about  a specific  project.  In  this  case,  the  Programming  Project 
Measurement  System  generates  a report  that  lists  the  answers  to 
the  questions  submitted  by  personnel  of  the  specified  project. 

The  second  type  of  query  is  a more  general  request  for  informa- 
tion. an  exam^e  of  which  is.  “What  has  been  the  utilization  of 
improved  programming  technologies  by  projects  in  the  data  base 
and  what  was  the  effect  on  project  productivity?”  The  answer  is 
provided  in  the  form  of  a listing  of  productivity  data  sorted  by 
project.  The  following  breakdown  indicates  the  types  of  report- 
ed and  derived  data  that  can  be  provided  in  response  to  a gen- 
eral inquiry: 

NO.  I ■ 1977  nOCSAMMINO  MtASUMMCNT  AND  ESTIMATION  S7 


3 


157 


TabU  2 Cborocfriiob—  of  pragraiM  in  rim  pfogramming  proinct  doM  boM 


A.  Small  less^omplex  systems 

• Batch  Stonge  and  retrieval 

• Batch  inventory 

• Batch  information  management 

• Batch  languages  preprocessor  and  information  management 

• Batch  reporting 

• Batch  financial  iafoiinaiioa 

• Batch  sdeniiflc  processing  simulatiaa 

• Batch  utility 

• Batch  openting  system  exerdser 

B.  Medium  les»<aiiipl«x  systtms 

• Spedal-piiipose  dau  management  (2) 

• Batch  stoiai^  and  retrieval  (2) 

• Process  control  simulation 

• Batch  reporting 

• Batch  data-base  utility  (2) 

• On-line  scientific  processing  simulation 

• Batch  on-line  scientific  infonnation  management 

• On-lins  busuteu  infotmatiott  management 

• On-line  storage  and  retrieval 

• Batch  hardware  teat  support 

• Batch  sdcntific  algorithm  feasibiiity  U) 

• Intenetive  scientific  processing  (2)  / 

• System  teat  support 

> Batch  planning  ( 3 ) 

• Batch  fflilitaiy  infonnation  management 

• Spedat-parpose  opentiag  system 

C.  Medinm  compies  systems 

• Rcai-thne.  spscdi-p«pote  system  exerciser 

• Spedat-purposa  operating  system  (2) 

• Batch  infotmatioo  modification 

• Batch  information  conversion 

• Data  managment 

• Sensor-based  missioa  control 

• On-line  scheduling 

> Sensor-baaed  miasioo  simulation 

• Interactive  scientific  processing 

• Process  control  ( 3 ) 

• On-line  graphics 

• System  performance  monitoring  and  measurement  (3) 

• Terminal  data  management 

• Interactive  infonnation  conversion 

• Operating  system  extensions 

D.  Large  complex  systems 

• Sensor-lMsed  mission  monitoring  and  control 

• Interactive  information  acquisition 

• Procass  control 

• Seneor-heeed  system  exerdser  (3) 

• leninr  timed  miaeinn  pmraiiini  end  -imniniiiiratinn  tTl 


Programming  project  data: 

• Number  of  lines  of  delivered  source  code  ordered  by  project. 
(Source  lines  are  80<haracter  source  records  provided  as 
input  to  a language  processor.  Job  control  languages,  data 
definitions,  link  edit  language,  and  comment  lines  are  includ- 
ed. Reused  code  is  not  included.) 
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TobU  3 Progromming  proi«cf  dMcnpHv*  dota  report  for  compUfpd  projpcn 


Mrdiun 

Qiiarliles 

SfKf 

25-75% 

Productivitv 

Source  lines  per  man  month  of  total 

;'4 

1. <0-440 

elfort 

Product 

Source  lines  (ihousendsi 

20 

10-59 

Percemage  of  code  lines  not  to  be 

5 

O-ll 

delivered 

Documentation  per  Uiousand  lines  of 

69 

27-167 

source  code  (pefss) 

Resource 

Computer  cost  ( as  a percentage  of 

18 

10-34 

project  cost) 

Toiai  elfort  I man  months) 

67 

37-186 

Avenge  manning 

6 

3.8-14.5 

Elfort  distnbution  of  preliminary 

18 

11-25 

design  review  ( percent) 

Distribution  of  ell^  (percent) 

Management  and  administnuion 

18 

12-20 

Analysis 

18 

6-27 

Programming  and  ttesign 

60 

50-70 

Other 

4 

0-6 

Duration  ( momhs) 

II 

8-19 

Deveiopnieni  error  detection 

Errors  per  thousand  source  lines 

3.1 

0.8- 8.0 

Incorrect  function 

76 

50-86 

Omitted  function  (percent) 

8 

0.22 

SUfiaurpntmi  tunaion  (percent) 

17 

7-25 

Enon  per  progammiiig  man  month 

0.9 

0.3 -2.4 

• Pages  of  documentation  (including  program  source  listings) 
delivered. 

• Source  languages  used  to  develop  code. 

Resource  data: 

• Total  etfort  (Ln  man-months,  including  management,  adminis- 
tration, analysis,  operational  support,  documentation,  design, 
coding,  and  testing)  required  to  produce  the  lines  of  source 
code  by  project. 

• Duration  of  project  in  months. 

Use  of  improved  programming  technologies  (expressed  as  a 

percentage  of  code  developed  using  each  technique) : 

• Structured  programming. 

• Top-down  development. 

• Chief  programmer  team. 

• Design  and  code  inspections. 
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Tobl*  4 Doto  fqr  cofiipi«t#d  54rv(C«  proj^cH 


Median 

J09f 

Qiianiles 

25-7f% 

Product 

Lines  of  maintained  source  code 

103 

56-474 

1 thousands) 

Ratio  of  developed  to  maintained 

0.04 

0-0.19 

code 

Resources 

Averate  tnanamg  per  project 

6 

4-11 

Maintained  source  lines  per  man 

IS 

S-24 

(thousands) 

Distributioa  of  effort 

ManafCinent  and  adimnistraiion 

IS 

10-20 

(percent) 

Analysis  (percent) 

10 

4-30 

Programming  (percent) 

72 

40-80 

Other  ( percent) 

3 

0-7 

Duration  (inonthsi 

18 

11-31 

Errors  detected 

Errors  per  thousand  lines  of  maintained 

1.4 

0.2 -2.9 

code 

Incorrect  function  (percent) 

73 

Omitted  fiinctioo  (percent) 

II 

Misinterpreted  fiiiKtion  (percent) 

13 

TabI*  5 Opicript>»«  doto  froio  on  going  projocN 

Median 

Quartiles 

Prograntming 

50% 

23  - 73% 

Source  lines  (thousands) 

12.3 

3.4-30 

Perceniage  of  code  lines  not  to  be 

2.6 

O-ll 

delivered 

Effort  (man  months) 

72 

30-203 

Distribution  of  effort: 

System  analysis  ( percent) 

15 

10-20 

System  design  (percent) 

20 

13-23 

Code  and  unit  test  (percent) 

30 

20-33 

Integration  and  test  (percent) 

20 

13-30 

Other  ( percent) 

5 

0-10 

Duration  (months) 

12 

9-18 

Median 

Quartiles 

Service 

50% 

25-75% 

Maintained  source  lines  (thousands) 

148 

55-340 

Ratio  of  developed  to  maintained  code 

0.03 

0-0.09 

Effort  (man  months) 

88 

27-183 

Average  maiming  (persons) 

3 

3-19 

Duration  (months) 

10.3 

6.3-  12 

I . 
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Derived  data: 


• Productivity  achieved,  ordered  by  project.  Programming 
productivity  is  defined  as  the  ratio  of  the  delivered  source 
lines  of  code  to  the  total  effort  (in  man-months)  required  to 
produce  the  lines  of  code,  and  is  computed  from  product  and 
resource  data. 

• Average  number  of  people  required  to  work  on  the  project, 
computed  by  dividing  the  total  effort  in  man-months  by  the 
duration  of  the  project  in  calendar  months. 

More  complex  and  extensive  search  and  analysis  questions  can 
also  be  answered.  These  are  supported  by  a question  analysis 
subsystem  of  the  Programming  Project  Measurement  System, 
which  incorporates  a statistical  package  for  the  manipulation 
and  statistical  analysis  of  many  types  of  data.  The  more  complex 
requests  can  be  grouped  into  two  types,  descriptive  and  analyti- 
cal. A descriptive  request  requires  searching  the  data  base  for 
specific  variables  or  derived  variables  and  computing  character- 
istics about  their  distributions  such  as  the  mean,  mode,  and  stan- 
dard deviation,  in  order  to  prepare  the  report.  Table  3 illustrates 
one  such  report  for  the  completed  projects  in  the  data  base. 

Because  of  the  variability  in  the  measurement  data,  the  statistics 
in  Table  3 are  presented  in  terms  of  medians  and  quaitilesttpie 
median  for  the  size  of  the  delivered  software  product  is  20  thou- 
sand lines  and  fifty  percent  of  the  projects  reported  that  the  sizes 
of  their  delivered  source  code  rang^  from  10  thousand  to  59 
thousand  lines.  The  effort  for  software  development  reported 
was  distributed  into  the  nuqor  categories  as  shown.  The  error 
detection  section  of  the  table  shows  the  distribution  of  errors 
reported  during  the  development  phase. 

Table  4 provides  descriptive  statistical  data  for  completed*ser- 
vice  projects.  Most  service  activity  is  not  purely  maintenance, 
but  includes  development  efforts  as  well.  The  ratio  of  developed 
source  lines  of  code  to  maintained  lines  of  code  was  4 percent  at 
the  median. 

Table  3 presents  data  on  on-going  programming  and  service 
projects.  Since  only  a small  percentage  of  these  projects  have 
been  completed  at  this  time,  the  statistics  represent  a mixture  of 
actual  measurements  and  estimates  at  various  stages  of  com- 
pletion. 


Productivity  anaiysla 

We  have  identified  five  nuyor  parameters  that  can  help  program- 
ming project  personnel  make  estimates.  These  paiameten.  pro- 
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ductivity,  schedule,  cost,  quality,  and  size,  are  listed  in  order  of 
increasing  difficulty  and  complexity  of  analysis.  Some  of  the 
difficulties  arise  from  a lack  of  detailed  data  in  the  data  base,  as  in 
the  case  of  schedule  data.  Complexity  of  the  quantitative  data 
can  create  other  difficulties.  One  significant  difficulty  is  in  iden- 
tifying and  measuring  independent  variables  that  can  be  used  to 
estimate  the  desired  variable,  as  is  the  case  in  estimating  the  size 
of  the  product  to  be  delivered. 

Productivity,  which  can  be  defined  in  terms  of  the  quantitative 
measures  that  are  in  the  data  base,  is  a vital  factor  in  ^ software 
estimating  processes  and.  therefore,  is  of  immediate  value  to 
project  personnel.  For  this  reason,  the  analysis  performed  to 
date  has  focused  on  productivity  estimation.  Productivity  has 
often  been  defined  as  the  ratio  of  output  to  input.  Programming 
productivity  is  defined  here  as  the  ratio  of  the  delivered  source 
lines  of  code  tosu  to  the  total  effort  in  man-months  (MM)  re- 
quired to  produce  that  delivered  product. 

The  basic  relationship  between  delivered  lines  of  code  and  effort 


r 

is  shown  in  Figure  1.  Each  plotted  point  represents  the  data 

reported  by  a completed  project  in  the  data  base.  A number  on 

the  chart  indicates  a position  where  a number  of  dau  points  are  * ■ 

grouped  sufficiently  close  together  that  they  cannot  be  individu- 

ally  identified  on  tte  plot.  The  data  have  bran  plotted  in  the  log- 

log  domain  so  that  they  become  approximately  linear.  The  linear  i . 

a - 

coefficients  become  power  relationships  when  transformed  back 

Z/ 

i__d 1 1 1 

too  Ift  IQK  lOOft  \U 

to  the  original  domain  of  the  data.  Tlie  least  squares  fit  to  the 
data  as  plotted  in  Figure  1.  yields  the  result: 

ocuvmo  coof  (SouRCt  imcs) 


prortHCthrWy 


E - 5.2Z,*-*' 
where 

E * total  effort  in  man  months 
and 

L ~ thousands  of  lines  of  delivered  source  code. 

This  relationship  is  nearly  a first-power  (or  linear)  relatibnship 
between  effort  and  product  size.  The  dashed  lines  indicate  the 
standard  error  of  the  estimate  on  either  side  of  the  least  squares 
fit  To  identify  the  sources  of  scatter  or  variation  of  Figure  1. 
those  variables  that  are  related  to  productivity  have  been  inves- 
tigated. Preliminary  findings  have  led  to  the  development  of  a 
productivity  estimator  that  provides  an  on-line  capability  to 
support  proposed  as  well  as  on-going  projects.  A set  of  sixty- 
eight  variables  was  selected  from  the  diua  base,  and  those  vari- 
ables were  analyzed  to  determine  which  were  significantly  re- 
lated to  productivity.  Twenty-nine  of  the  variables  showed  a 
significantly  high  correlation  with  productivity  and  have  there- 
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fore  been  retained  for  use  in  estimating.  Table  6 lists  these  vari- 
ables and  the  reponses  associated  with  them.  To  illustrate  the 
meaning  of  Table  6.  consider  the  first  entry,  which  is  derived 
from  a multiple-choice  question  that  asks  the  information  sup- 
plier to  circle  his  response  to  the  following  statement:  Customer 
interface  is  (less  than,  equal  to.  greater  than  I normal  complexity. 

When  the  mean  productivity  was  computed  for  all  the  completed 
reports  in  the  data  base  that  indicated  less  than  normal  customer 
inteifKe  complexity,  the  result  obtained  was  500  delivered 
source  lines  of  code  per  man-rnonth  of  effort  (DSUMM).  By  a simi- 
lar computation,  the  mean  productivity  for  all  projects  that  re- 
ported normal  complexity  was  295  DSL/MM,  and  the  mean  pro- 
ductivity for  those  reporting  greater  than  normal  complexity 
experience  was  124  DiUMM.  The  change  in  productivity  be- 
tween less-than-normal  and  greater-thanniormai  customer  inter- 
face is  376  OSL/MM,  as  noted  in  the  final  column  in  Table  6.  Three 
variables  in  the  table  (oveiall  personnel  experience,  code  com- 
plexity, and  design  crmstrainu)  were  formed  by  combining  the 
answers  to  several  questions  in  the  questiormaire.  It  should  be 
noted  that  this  analysis  was  performed  on  each  variable  indepen- 
dently and  does  not  take  into  account  either  the  possibility  that 
these  variables  may  be  correlated,  or  that  there  may  be  inter- 
related effects  associated  with  them. 

The  twenty-nine  variables  were  then  combined  into  an  index, 
based  on  the  effect  of  each  variable  on  productivity,  as  indicated 
by  the  above  analysis,  to  form  a productivity  index.  The  produc- 
tivity index  is  computed  as  follows: 


i. 


r 


where 

/ » productivity  index  for  a project 

IV, « question  weight,  calculated  as  one-half  log,,  of  the  ratio 
of  total  productivity  change  indicated  for  a given  question  i 
X,  * question  response  (4-1.  0 or  —1),  depending  on  whether 
the  response  indicates  increased,  nominal,  or  decreased 
productivity 

An  index  was  computed  for  fifty-one  projects,  and  a plot  of  ac- 
tual productivity  for  each  project  versus  the  computed  produc- 
tivity index  and  the  least  squares  fit  to  this  relationship  is  shown 
in  Figure  2.  The  standard  error  of  the  estimate  (standard  devia- 
tion of  the  residuals)  is  shown  as  dashed  lines. 

To  support  project  estimates,  a shortened  version  of  the  data  rapM 
collection  form  is  used  that  contains  excerpted  questions  as-  mUri 
sociaied  with  the  twenty-nine  variables  used  in  the  index.  A 
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TaW*  6 VariabiM  that  camioM  iigaidwtty  mM«  pratrowilt  prarfwcfinty 


Question  or  Variable 

Response  Croup  > 

Mean  Froductivity 
iOSUMM) 

Froductivity 

Change 

(DSUMM) 

Customer  interface  ' 

Normal 

Normal 

> Normal 

complexity 

500 

293 

124 

376 

User  pankipuioii 

None 

Some 

Much 

in  the  dehmiioa  of 

491 

267 

203 

286 

requaemettls 

CustooMr  ofisiaawd 

Few 

Many 

pcofnm  dttisB  ctaBfM 

297 

196 

101 

Cuttomer  experience 

None 

Some 

Much 

with  the  appiicaiioa 

318 

340 

206 

112 

area  of  the  prpiect 

Overall  penoonel 

Low 

Avetape 

Hiph 

experience  and  qualili- 

132 

237 

410 

278 

catMm 

Percentape  of  pro- 

< 23% 

23  - 30% 

> 30% 

irnmi— a rtnini  rIeTil 

133 

242 

391 

238 

opmeni  who  partidpaied 

in  deaisn  of  ftinctitxtnl 

spcctflcsiioM 

Previoua  experience 

Miniinal 

Aveiape 

Extensive 

with  operaiianal 

146 

270 

312 

166 

conipiuar 

Prcvioiit  cxptricBCS 

Minimal 

Aveiape 

Extensive 

with  proetamminc 

122 

223 

383 

263 

Pmviova  experience  n 'th 

Miniinal 

Aveiape 

Extensive 

application  of  siiniiar  or 

146 

221 

410 

264 

mater  size  and  coot- 

plexity 

Ratio  of  avetase  staff  tine 

<0.3 

0.3 -0.9 

> 0.9 

to  duration  (peopie/inonih) 

303 

310 

173 

132 

Hardware  under  concumm 

No 

Yes 

deveiopment 

297 

177 

120 

Ocvelopment  computer 

0% 

1-23% 

> 23% 

access,  open  under  special 

226 

274 

337 

131 

request 

Development  computer 

0-10% 

11-83% 

> 83% 

cioMd 

303 

231 

170 

133 

Classilled  security  envi- 

No 

Yes 

ronmem  for  computer  and 

289 

136 

133 

25%  of  promw  *ad  data 

prognin,  nimung  in  the  Tune  Sharing  Option  of  OS/vs  (TSO)  was 
developed  to  compute  and  list  the  index  estimates.  This  terminai* 
based  prognun  allows  rapid  response  to  project  requests  for 
information.  The  estimate  of  expected  productivity  is  returned  to 
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Qutslion  or  yariabie 


Response  Group 
Mean  ProJui'tiviiy 
IDSL  MW» 


Productivity 

CImnve 

iDSLiMMi 


I 


i 

i: 

Structured  programming 

0-33% 

169 

34-66% 

66% 

301 

132 

Design  and  code  inspec- 
tiofia 

0-33% 

220 

34-66% 

300 

> 66% 

339 

119 

1. 

Top  down  dnvgiopiiiwu 

0-33% 

196 

34-66% 

237 

>66% 

321 

125 

1. 

Chief  pragrumer  tenm 
usage 

0-33% 

219 

34-66% 

>66% 

408 

189 

Ovcnll  comptexicy  of 
cod*  develop 

< Average 
314 

> .Average 

185 

129 

{ . 

Complexity  of  applicatioa 
processiiig 

< Average 
349 

Average 

345 

> Avenge 

168 

181 

Complwaty  of  pratnm 
flow 

< Average 
289 

Avenge 

299 

> Average 
209 

80 

1: 

Ovendl  coniiniaa  on  pro* 
gram  dmign 

Mimmal 

293 

Average 

286 

Seven 

166 

107 

\ 

Program  design  constraints 
on  main  storage 

Minimal 

391 

Avenge 

277 

Severe 

193 

198 

Progimm  dMigtt  coostnintt 
on  doling 

Minimal 

303 

Avenge 

317 

Seven 

171 

132 

i 

y 

i 

<*  0 

Code  for  rcat-time  or  inter- 
active operaiioa.  or  exe- 
ctifing  under  mv«i»  daiag 
conatiaim 

< 10% 
279 

10-40% 

337 

>40% 

203 

76 

Percentage  of  code  for 
delivery 

0-90% 

159 

91-99% 

327 

100% 

265 

106 

r 

I. 

Code  ciaaaiiled  as  non-  0-33% 

matheinaticai  application  and  188 
t/O  formatting  programs 

34-  66% 
311 

67-100% 

267 

79 

1- 

Number  of  classes  of  items 
in  the  data  base  per  1000 
lines  of  code 

0-15 

334 

16-80 

243 

A - 

141 

i • 

r 

Number  of  pages  of  de-  0-32 

livered  documentation  per  320 

1 000  lines  of  delivered  code 

33-88 

252 

> 88 

195 

125 

, I 


the  requester  in  the  form  of  a report  that  contains  a comparison 
between  the  project  estimate  and  the  one  derived  from  the  data 
base.  Also  included  is  a list  of  the  reported  attributes  or  variables 
that  had  a signiftcant  influence  on  the  estimate.  Where  possible, 
detailed  discussions  are  held  on  special  factors  associate  with  a 
project  that  may  not  be  properly  handled  in  the  present  algorithm. 
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Figur*  2 Rtlatiofithip  b«tw*«n  pr«dtictivt>y  and  prodwaivity  ind«»  for  twontyonino 
vpriobloc 


ngliro  3 CrtnoMd  prodoctMty  lor  o hypothotkol  projotf 
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Figure  3 is  a plot  similar  to  the  one  shown  in  Figure  2.  which  is 
presented  for  a hypothetical  prcject.  The  productivity  index  is 
computed  for  the  project  from  responses  to  the  proposal  ques* 
tionnaire  and  yields  the  expected  productivity  to  be  attained,  as 
determined  by  the  measurements  data  base.  In  the  case  shown 
in  Figure  3,  the  estimated  productivity  is  seen  to  be  two  hundred 
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Tabi*  7 A^4itioiiol  productivity  roiotod  voriobi** 


Vuriuble 

Response  Chunitu 
( low  to  high 
pertenii 

Proiim-iivin 

Change’ 

( percenil 

Ratio  of  JevelopeC  to 

705 

onpiuJ  plus  Jevelopcd 
code 

U to  lOU 

Elibn  at  pnmary  devdop- 
iMm  locatiM 

50  to  too 

:i5 

Rcmott  job  entry  computer 
access 

0 to  too 

205 

•9mmi  uo  icMi  ’UPMfti'  tit  to  iteta  m Fi|af«  4. 


delivered  source  lines  of  code  per  tnan-month  (OSUMM).  with 
one  standard  error  range  of  1 15  to  340  osUMM.  The  project 
team's  independently  developed  productivity  estimate  for  the 
same  conditions  was  150  DSUMM.  Thus,  in  this  case,  the  project 
estimate  is  a more  conservative  estimate  than  that  given  by  the 
productivity  index. 

Consider  additional  conclusions  that  can  be  drawn  from  Figure  otner 
3.  If  we  assume  a normal  distribution  for  the  observations,  when  MUmatM 
they  are  plotted  as  a log  of  the  productivity  versus  a log  of  the 
productivity  index  in  Figure  3.  the  probability  that  the  proj- 
ect would  have  a productivity  estimate  in  region  A (i.e..  less 
than  2.06  or  the  log,,  of  1 1 5 DSUMM ) is  about  0. 1 7,  The  proba- 
bility that  the  productivity  estimate  would  be  in  region  B (i.e.. 
between  2.06  and  2.30.  the  log  of  1 1 5 and  200  DSUMM ) is  about 
0.33.  Similarly.  is  0.33  and  P^  is  0.17.  This  is  the  probability 
distribution  of  productivity  estimates,  not  the  cumulative  proba- 
bility that  a project  will  (or  will  not) achieve  or  exceed  the  pro- 
ductivity that  was  estimated. 

investigation  is  continuing  into  other  variables  from  the  data 
base  that  may  also  be  related  to  productivity.  Figure  4 shows 
several  distributions  that  appear  to  have  a significant  relation- 
ship to  productivity,  although  in  two  of  these  cases  they  are 
based  on  a limited  number  of  observations.  Table  7 expresses 
the  net  effect  of  the  data  plotted  in  Figure  4 in  tabular  form. 

Figure  4(A)  shows  productivity  (in  source  lines  of  code  per 
man  month)  plotted  against  the  ratio  of  developed  source  code 
to  the  sum  of  any  original  (or  reused)  code  plus  the  developed 
code.  The  plot  in  Figure  4(A)  suggests  that  productivity  is  high- 
est when  there  is  no  originai  or  reused  code,  that  is.  when  ail  the 
code  is  developed  from  the  inception  of  the  project.  As  the  per- 
centage of  reused  code  grows,  the  expected  productivity  de- 
creases. 
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Figure  4(B),  although  it  contains  a large  amount  of  scatter,  sug- 
gests that  when  the  development  effort  is  spread  across  more 
than  one  location,  i.e..  as  the  percentage  of  effort  at  the  primary 
location  becomes  less  than  100  percent,  the  productivity  de- 
creases. Another  question  currently  of  interest  is  the  impact  of 
remote  job  entry  on  productivity.  Most  of  the  completed  proj- 
ects in  the  data  base  were  developed  without  the  use  of  termi- 
nals. as  Figure  4(C)  shows.  On  the  basis  of  a least  squares  fit. 
however,  those  projects  that  use  remote  job  entry  do  appear  to 
have  an  increase  in  productivity. 
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other  results  of  programming  analysis 

Although  the  primary  effort  has  been  directed  toward  productiv- 
ity analysis,  other  analyses  have  been  performed  on  the  data 
base.  Results  of  these  efforts  to  the  present  time  are  presented 
here.  The  data  can  be  used  to  check  productivity  estimates,  and 
to  check  current  project  parameters  against  past  experience,  as 
reflected  by  the  dau  base.  Such  results  provide  a multidunen- 
sional  approach  to  crosschecking  a number  of  the  factors  that 
enter  into  estimates  of  effort:  productivity,  duration,  documenta- 
tion. and  computer  costs.  These  results  also  indicate  the  nature 
of  the  analyses  that  can  be  performed  against  the  data  base. 

Documentation  is  a critical  product  of  every  software  project, 
and  documentation  costs  are  an  important  component  of  the  es- 
timation process.  A useful  parameter  for  measuring  documenta- 
tion is  number  of  pages.  Figure  5 is  a plot  of  delivered  documen- 
tation in  number  of  pages  versus  delivered  source  lines  of  code. 
Documentation  is  defined  here  as  program  functional  specifica- 
tions and  descriptions,  users'  guides,  test  specifications  and  re- 
sults. flow  charts,  and  program  source  listings  that  are  delivered 
as  part  of  the  documentation.  As  a first  approximation,  the  least 
squares  fit  indicates  that  a linear  or  first-order  relationship  ex- 
ists; that  is.  the  number  of  pages  of  delivered  documentation 
varies  directly  as  the  number  of  lines  of  source  code. 
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After  programming  project  estimates  have  been  completed, 
those  estimates  can  checked  against  the  dau  base  by  using 
the  plots  in  Figures  5- 10.  If.  for  example,  the  size  of  the  deliv- 
ered software  product  is  estimated  as  ten  thousand  lines  of 
source  code  (as  shown  in  Figure  5)  it  can  be  seen  ftom  past 
experience  that  the  expected  number  of  pages  of  documenution 
to  be  delivered  is  five  hundred.  The  range  for  one  standard  error 
for  this  given  value  is  one  hundred  eighty  to  thirteen  hundred 
pages.  This  provides  an  independent  calibration  point  that  the 
manager  can  use  to  compare  his  estiiiuue  against  the  experience 
of  past  projects.  A significant  difference  between  the  two  does 
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not  necessarily  imply  an  error  on  the  pan  of  the  manager,  but  it 
does  suggest  that  the  assumptions  and  estimates  might  be  re- 
examined. 


The  question  of  how  much  time  to  allow  for  the  development  of 
software  is  always  diificult  to  assess.  The  relationship  between 
duration  (expressed  in  months)  and  delivered  source  lines  of 
code  is  shown  in  Figure  6.  Project  duration  as  a function  of  total 
effort  in  man  months  is  shown  in  Figure  7.  initial  analysis  indi- 
cates that  a cubic  relationship  fits  the  data  in  both  of  these  fig- 
ures. This  implies  that  the  duration  of  effort  increases  by  the 
cube  root  of  the  number  of  source  lines  of  code  delivered  or  by 
the  cube  root  of  the  total  effort  applied  to  the  development  of 
the  code.  For  example,  for  a project  that  is  developing  a soft- 
ware product  of  10  thousand  lines  of  source  code,  the  expected 
duration  of  the  effort  is  4.0  x lO.O"'^  or  9.6  months.  Figure  7 
does  not  imply  that  simply  reducing  total  effort  automatically 
permits  a reduction  in  project  durab'on.  Such  a reduction  would 
more  likely  make  it  impossible  to  produce  and  test  the  required 
volume  of  code. 


project 

duration 
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The  staff  size  utilized  to  develop  a given  software  product  is  staff 

influenced  by  a number  of  factors,  including  the  time  allowed  for  atae 

devdopmeoL  the  amount  of  code  to  be  developed,  and  the  staff- 
ing rates  that  can  be  achieved.  After  a project  has  been  estimat- 
ed. one  convenient  measure  used  to  describe  the  size  of  the  proj- 
ect is  the  average  number  of  people  required.  Figure  8 shows  a 
relationship  that  can  be  used  as  another  check  on  the  estimating 
process.  It  shows  the  relationship  between  the  staff  s:ze~ex- 
pressed  in  terms  of  the  average  number  of  people  (defined  as 
total  man-months  of  effort  divided  by  the  duration)  -and  the 
total  effort  applied. 


Estimating  computer  costs  is  very  difficult,  but  at  the  same  time 
it  can  also  be  a very  significant  fraction  of  the  total  cost.  Al- 
though only  eighteen  of  the  completed  projects  in  the  dau  base 
had  computer  costs  reported,  some  interesting  relationships  are 
indicated  when  computer  cosu  are  compared  with  the  amount  of 
delivered  code  and  the  total  effort,  as  is  shown  in  Figures  9 and 
10.  In  Figure  9.  two  observations  (circled)  are  evidently  out  of 
bounds  when  plotted  against  delivered  code.  These  same  two 
observations,  however,  fit  well  with  total  effort,  as  shown  by  the 
plot  in  Figure  10.  Based  on  this  limited  evidence,  it  appears  that 
computer  cosa  are  closely  related  to  effort,  and  they  appear  to 
have  nearly  a first  power  (or  linear)  relationship.  Note  that  in 
Figure  9.  the  two  out-of-bounds  points  are  not  included  in  deter- 
mining the  least-square  fit. 
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The  present  approach  to  productivity  estimation,  although  use- 
ful. is  far  from  being  optimized.  Based  on  the  results  of  the  vari- 
able analysis  described  in  this  paper,  and  supplemented  by  the 
results  of  the  continued  investigation  of  additional  variables  re- 
lated to  productivity,  an  experimental  regression  model  has  been 
develop^  Preliminary  results  indicate  that  the  model  reduces 
the  scatter.  Further  work  is  being  done  to  determine  the  potential 
of  regression  as  an  estimating  tool,  as  well  as  to  extend  the  anal- 
yst of  the  areas  of  computer  usage,  documentation  volume, 
duration,  and  staffing. 


AppomUx 

The  effective  utilization  of  programming  measurements  data 
requires  the  ability  to  store,  retrieve,  process,  and  report  data. 
Specialized  capabilities  to  do  various  types  of  statistical  analy- 
ses are  also  required.  These  capabilities  are  provided  by  a Pr^ 
gramming  Project  Measurement  System.  This  system  is  com- 
posed of  two  subsystems,  the  question  processing  subsystem 
and  the  question  analysis  subsystem.  The  basic  functions  pro- 
vided by  the  question  processing  subsystem  are  the  maintenance 
of  the  data  b^  (which  contains  the  information  submitted  in 
response  to  the  questionnaire),  the  retrieval  and  listing  of  data 
from  the  data  base  in  various  report  formau.  and  the  extraction 
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of  data  for  transfer  to  the  question  analysis  subsystem  for  statis- 
tical analysis.  Figure  1 1 shows  the  overall  flow  of  information  in 
the  programming  project  measurement  system.  The  question 
analysis  subsystem  uses  the  Statistical  Package  for  the  Social 
Sciences  (a  product  of  SPSS.  Inc.),  which  is  an  integrated  sys- 
tem of  computer  programs  for  the  analysis  of  data,  and  provides 
the  user  with  a large  set  of  procedures  for  data  selection,  trans- 
formation. and  file  manipulation,  and  offers  a large  number  of 
commonly  used  statistical  routines. 
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Statistical  routines  include  descriptive  statistics,  frequency  dis- 
tribudons,  cross  tabuladons,  correladon.  partial  correlation,  mul- 
dpte  regression,  and  factor  analysis.  The  package  has  its  own 
internal  data  management  facilities  that  can  be  used  to  modify 
analysis  files  of  data  and  can  be  used  in  conjuncdon  with  any  of 
the  stadsdcal  procedures.  These  facilides  enable  the  user  to 
generate  variable  transformadoos,  recode  variables,  sample,  se- 
lect or  weight  specified  cases,  and  add  to  or  alter  the  data  or  the 
analysis  files. 
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Project  data  enter  the  Programming  Project  Measurement  Sys- 
tem by  way  of  quesdonnaires  that  ate  answered  by  project  per- 
sonnel. At  the  incepdon  of  the  measurement  program  discussed 
in  this  paper,  one  quesdonnaire  was  used  for  both  development 
and  service  (maintenance)  contracts.  On  service  contracts, 
quesdonnaites  were  to  be  submitted  quarterly.  For  development 
projects,  four  quesdonnaites  were  to  be  prepared  by  the  project 
at  m^r  milestones  during  the  life  of  the  project.  Idendcal  ques- 
donnaires were  to  be  submitted,  but  not  every  item  requited  an 
answer  at  each  submission.  The  four  reporting  milestones  were 
the  following: 


Start  of  work. 

Preliminary  design  review  or  equivalent. 

Top-down  programming— completion  of  integration  of  one- 
half  of  the  program  units,  or 

Bonoffl-up  programming— completion  of  unit  test  of  three 
quarters  of  the  program  units. 

Acceptance  test  completion. 


Problems  arose  when  project  personnel  tried  to  use  the  same 
quesdotmaire  form  for  both  development  and  service  contracts; 
dilferetKes  between  those  two  types  of  activities  made  it  difficult 
to  fit  all  the  necessary  questions  into  one  questiotuiaire  format. 
A further  problem  was  the  reporting  frequency  of  development 
contracts.  The  four  milestones  might  often  be  six  months  to  two 
years  apart,  and  many  changes  could  occur  in  project  organiza- 
tion. in  project  specifications,  and  in  the  definitions  of  products 
to  be  delivered,  so  that  it  was  difficult  to  correlate  questionnaire 
responses  from  milestone  to  milestone. 
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For  these  reasons  changes  were  made  in  the  questionnaires  and 
the  frequency  of  reponing.  Separate  questionnaires  were  created 
for  development  projects  (Software  Development  Reports)  and 
for  service  efforts  (Software  Service  Reports).  Development 
reports,  which  cover  detailed  qualitative  items  as  well  as  quanti- 
tative data,  are  submitted  at  the  start  of  work  and  again  at 
acceptance  test  completion.  Between  these  two  submittals,  a 
Monthly  Software  Development  Report  is  submitted.  This  is  a 
one-page  summary  of  the  status  of  a product,  cost,  and  effort 
that  is  submitted  each  month.  The  Software  Service  Report  is  an 
overview  of  a product  that  is  being  serviced  and  is  submitted  at 
the  start  and  end  of  service.  The  Quarterly  Software  Service 
Report  is  a summary  of  the  product,  cost,  and  effort  status,  plus 
a detailed  reporting  of  errors  and  their  impact.  Reporting  is  done 
by  programming  projects  that  are  developing  or  servicing  prod- 
ucts in  the  form  of  lines  of  code  and  that  employ  two  or  more 
programmers  with  an  expenditure  of  twelve  or  more  man- 
months  of  effort. 
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ABSTRACT ; A great  numcer  of  the  cturrent  problems  in  software  product  life 
cyle  management  can  be  traced. to  a lack  of  understanding  of  the  design  process 
in  general,  and  specifically  with  respect  to  software.  Because  this  workshop 
session  is  concerned  with  resource  allocation  and  management,  these  topics  are 
somewhat  emphasized  in  the  discussion  that  follows.  However,  the  central  themes 
are  that  the  proper  understanding  of  the  design  process  should: 

(a)  Significantly  enhance  the  ability  to  break  a design  into  parts  which 
can  be  independently  developed. 

(b)  Significantly  reduce  the  amount  of  both  manpower  and  time  required  to 
coa^lete  a design,  including  .ic^ lamentation  and  installation. 

(c)  Significantly  enhance  the  ability  of  a design  to  maintain  its  integrity 
in  the  fact  of  functional  changes,  both  during  originaul  design  efforts, 
and  subsequently  during  the  product  evolutionary  phases. 

X X X X X X X 

What  is  a "design?"  What  is  the  difference  between  a design  and 
its  realization?  What  makes  one  design  "better  than  another?" 

What  makes  two  designs  similar?  What  is  or  are  the  central  prob- 
lam(s)  of  the  design  process?  what  is  the  "design  process?"  How 
do  we  partition  a design  into  "man-sized  chunks?"  When  does  the 
design  process  start,  and  when  does  it  stop?  What,  if  any,  differ- 
ences exist  between  specifications  and  a design? 

These  emd  others  are  truly  the  basic  questions  we  deal  with  when 
we  look  at  the  software  life  cycle  and  attempt  to  manage  it.  Yet, 
These  questions  have  seldom  if  ever  been  directly  faced,  and  so 
seldom  if  ever  solved  satisfactorily  - certainly  not  theoretically, 
and  only  occasionally  in  practice.  The  theme  of  this  paper  is  the 
need  for  a small  and  practical  oriented  research  effort  to  answer 
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these  questions  in  the  context  of  the  few  basic  philosophical 
approaches  known  in  both  software  and  normal  engineering,  and  to 
define. a set  of  experiments  which  can  differentiate  the  character- 
istics of  the  processes  developed. 

Although  design  is  perhaps  the  most  characteristic  activity  of  a 
technological  civilization,  it  is  perhaps  the  least  understood  and 
certainly  the  least  recognized  and  researched  of. mam's  technical 
activities.  Perhaps  this  is  because  we  all  do  it  - it  is  too 
normal  of  an  activity  to  warrant  serious  attention.  In  those  engin- 
eering arts  and  sciences  based  on  the  natural  physics,  traditions 
dating  back  two  centuries  give  an  appearance  of  understanding.  And, 
these  traditions  do  result  in  designs  which  work.  Zven  here,  how- 
ever, no  significant  literature  exists  which  directly  addresses  the 
basic  questions  posed  above.  In  software  engineering,  the  problem 
of  design  is  more  sharply  in  focus  because  we  clearly  do  not  have 
traditions  that  work.  As  in  other  areas  between  software  engineer- 
ing and  the  natural  engineering,  it  is  also  probably  true  that  the 
knowledge  to  be  gained ' from  an  exploration  and  understanding  of 
software  designs  and  the  design  processes  can  significantly  contri- 
bute to  am  understanding  of  the  natural  engineering  design  process. 

The  vein  of  practical  value  in  an  exploration  of  the  idea  of  "what 
is  a design,"  and  "what  is  the  design  process"  is  unimaginably  rich. 
The  entire  field  of  computer  measurement  and,  more  recently,  soft- 
ware physics  was  a direct  result  of  mining  these  ideas.  Beyond 
that,  clear  relationships  exist  to  mamy  if  not  all  of  the  fundamental 
problems  that  currently  plague  software  engineering  and  life  cycle 
management . 

The  starting  point  for  the  discussion  is  to  make  a clear  distinction 
between  a design  and  a realization  of  a design  in  some  physical 
media.  Take  for  example  an  actual  automobile  and  a set  of  engineer- 
ing drawings  from  which  the  specific  automobile  was  constructed. 


The  set  of  engineering  drawings  is  the  design,  and  the  specific 
autw mobile  is  one  of  the  realizations  of  the  design.  Many  reali- 
zations niay  exist:  only  one  design  exists.  A second,  and  perhaps 
more  illustrative  example  is  the  set  of  ONA  chromosomes  any  animal 
or  plant  carries,  and  which  is  the  determining  pattern  or  design  for 
that  living  thing.  If  we  consider  dogs,  we  Jcnow  that  many  distinctly 
different  breeds  can  be  defined  by  their  chromosomes,  yet  the  essen- 
tial "dogness"  of  each  breed  is  not  disturbed  because  inter-breeding 
is  possible.  Thus,  we  can  consider  dogs  as  having  a design  or 
pattern  inherent  in  their  chromosomes  which  defines  "dogness,"  and 
sub-designs  for  different  breeds  which  are  like  options  and  differ- 
ent models  of  the  same  automobile.  This  thought  can  be  pushed  much 
further,  and  results  in  the  concept  of  "key  structures"  in  a design. 
More  on  this  point  is  considered  later. 

The  important  point  in  the  above  discussion  is  that  a design  is  a 
pattern,  from  which  a physical  realization  can  be  obtained.  Clearly 
the  existence  of  a DNA  pattern  represents  a design  in  this  sense. 
Eqraally  clearly,  DNA  alone  is  insufficient  to  actually  produce  a 
living  entity:  an  environment  and  set  of  processes  (forces  capable 
of  using  energy  to  cause  state  changes)  must  be  available  by  which 
to  cause  some  physical  material  to  assume  the  states  defined  by 
the  pattern  or  design.  On  reflection,  this  is  true  of  any  design 
which  is  to  be  used  to  "generate"  one  or  more  realizations.  That 
is,  a design  not  only  specifies  a set  of  states,  but  is  only  valid 
in  the  context  of  some  set  of  processes.  We  shall  call  this  set 
of  processes  the  realization  environment.  For  humans,  the  realiza- 
tion environment  is  provided  in  the  womb.  For  cars,  by  a sat  of 
factories,  mines,  and  power  generating  equipment. 

It  is  most  important  to  realize  that  for  software,  the  realization 
form  is  a set  of  physical  states  in  storage  along  with  processors 
capable  of  changing  these  states  by  executing  instructions.  That 
is,  the  lines  of  code  written  on  paper,  and  the  data  forms  similarly 
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defined,  represent  the  design,  not  the  realization  environment  to 
be  realized:  this  may  be  a compiler,  assembler,  loader,  bookstrap 
loader,  or  other  mechanism  - software  or  hardware  - but  the  design 
must  be  written  in  its  context. 

But  if  a line  of  code  is  to  be  considered  as  part  of  a design,  then 
coding  is  part  of  the  design  process.  What  then  of  the  rest  of  the 
process?  And  clearly,  the  code  and  data  form  specifications  alone 
cannot  be  the  only  form  in  which  a software  design  can  be  stated: 
"higher  level"  designs  are  known  to  be  possible  as  well.  Both  , 

engineering  drawings  and  DNA  provide  examples  which  clarify  this 
point,  and  which  are  equally  valid.  However,  engineering  drawing  ^ 
trees  are  more  familiar  and  visual,  so  they  will  be  used. 

An  engineering  drawing  tree  is  a set  of  drawings  which  can  be  visual- 
ized either  as  an  upside  down  t. ae  or  a pyramid.  At  the  top  of  the 
pyramid  is  a single  drawing:  typically  a three-view  drawing  of  the  ..i 
entire  thing  being  designed.  If  an  aircraft  is  being  designed,  it 
shows  the  complete  airplane.  At  the  lowest  level  of  design  one  has 
"detail  parts"  or  "purchased  sub-assemblies."  In  the  latter  instance, 
these  affectively  represent  additional  engineering  drawing  trees 
that  ultimately  all  end  up  at  the  "detail  part  level."  In  between 
the  master  lines  and  dimensions  drawing  (at  the  apex  of  the  pyramid) 
lie  many  levels  of  drawings  called  "assembly  drawings."  The  ques- 
tions of  concern  are:  (1)  what  does  an  assembly  drawing  represent, 
and  (2)  what  distinguishes  a detail  part  drawing  from  an  assembly 
drawing. 


A detail  part  is  the  lowest  level  of  design,  that  is  clear.  But 
how  does  a designer  know  to  identify  the  part  as  a detail  part, 
rather  than  as  an  assembly  drawing?  The  answer  lies  in  the  designer's 
knowledge  of  the  realization  environment.  A drawing  is  called  a 
detailed  part  drawing  when  a set  of  processes  exist  which  are  known 
to  be  capable  of  realizing  the  drawing  design  without  further  infor- 
mation . 


A line  of  code  in  some  language,  with  its  operand  data  forms,  will 
be  realized  in  storage  without  further  information  being  needed. 

It  therefore  is  equivalent  to  the  concept  of  a "detailed  part"  in 
engineering  drawing  terms.  The  general  term  I have  defined  for  a 
"detailed  part"  level  software  design  statement  is  an  "operational 
relationship,"  since  it  may  be  a subroutine  call  (equivalent  tc  a 
"purchased  sub- assembly") , a con^lete  sort  program,  or  a source 
code  statement  in  some  language  along  with  its  operand  data  forms. 

What  is  an  "assembly  drawing,"  which  is  typically  made  up  of  "sub- 
assembly" and  detail  parts  drawings?  Ultimately,  of  course,  any 
design  is  defined  in  terms  of  detail  part  drawings  when  it  is  com- 
pleted, but  even  then  the  assembly  drawings  are  considered  part  of 
the  design  engineering  drawing  trees.  An  understanding  of  the 
answer  is  crucial  to  the  entire  concept  of  hierarchic  design;  an 
assembly  drawing  represents  an  entity  which  is  a complete  design 
statement  at  the  level  of  detail  of  the  entity,  but  which  must  be 
further  specified  to  be  capable  of  realization.  The  assembly  draw- 
ing shows  the  relationships  of  the  parts,  additional  sub-assembly 
amd/or  detail  part  drawings  are  needed  to  specify  the  parts  at  a 
realizable  level  of  detail.  At  a given  level  of  detail,  the  design 
is  completely  specified  by  the  corresponding  assembly  drawings 
even  though  the  level  of  detail  chosen  may  not  be  at  the  level  of 
realization.  Further,  and  of  considerable  importance  to  software 
design,  each  asscunbly  drawing  when  decomposed  will  never  (a)  intro- 
duce new  parts  not  visualized  at  the  assembly  level,  nor  (b)  inter- 
face with  other  assemblies  not  visualized  at  the  assembly  level. 
Generally,  neither  condition  is  met  by  software  designs  represented 
by  hierarchies  of  block  diagrams  and  flow  charts. 

One  of  the  basic  flaws  in  the  software  design  processes  currently 
practiced  is  illuminated  by  the  properties  of  engineering  drawing 
trees,  and  particul2u:ly  the  assembly/sub-assembly/ detail  part  dis- 
cussion above.  That  is,  levels  of  design  statements  are  impossible 


unless  the  design  notation  is  such  that  (a)  a design  decision  at 
one  level  can  be  detailed  without  introducing  new  interfaces,  and  *• 

(b)  that  more  detailed  design  decisions  can  be  mapped  directly  and  > 
uniquely  back  to  a higher  level  decision.  The  problem  is  not  »• 

simply  a notational  problem.  The  problem  is  fund2unentally  inherent 
in  the  view  that  software  must  be  designed  as  a sequential  set  of 
operations.  That  is,  it  cannot  be  generally  solved  by  any  method 
which  describes  the  design  of  the  software  as  a sequential  entity. 

The  reason  for  this  general  negation  of  a sequential  view  of  soft- 
ware for  design  purposes  can  be  seen  directly  by  considering  the 
"central  question  of  design."  It  may  be  phrased  in  several  differ- 
ent ways,  each  illuminating  certain  aspects  of  the  problem.  The 
most  general  way,  however,  is  simply  "how  can  a structure  or  entity 
exhibit  properties  which  are  not  inherent  in  any  of  its  substruc- 
tures." For  example,  how  can  an  airplane  fly  when  none  of  its 
parts  can  fly?  Part  of  the  answer  to  the  central  question  lies  in 
the  basic  description  of  the  design  process;  i.e.,  the  design 
process  is  the  process  of  realizing  overall  properties  by  replacing 
functional  requirements  with  structures  and  their  relationships. 

More  coxnmonly,  this  is  phrased  as  "replacing  function  with  form," 
but  this  latter  phrasing  fails  to  explicitly  mention  that  it  is 
not  simply  structure  which  is  being  specified,  but  relationships 
between  structures  as  well.  It  is  precisely  this  point  which  cuts 
to  the  heart  of  the  central  design  question,  because  some  properties 
arise  from  relationships > and  others  are  inherent  to  substructures 
and  are  transmitted  to  the  higher  level  structures.  Airplanes  fly, 
the  parts  do  not.  The  property  of  being  able  to  fly  arises  from 


the  shapes  and  other  relationships  of  the  parts.  However,  the  mass 
of  the  airplane  is  simply  the  sum  of  the  mass  of  each  of  its  parts. 

The  central  question  of  design  has  its  implications  on  the  software 
design  process,  and  particularly  on  the  method  of  representing  a 
design  notationally.  In  this  context,  the  question  is  what  struc-  ! j 
tures  to  select  to  bring  forth  an  entity  with  the  desired  functional 


proper-ties;  i.e.,  that  will  interact  with  the  presumed  environment 
of  any  realization  of  the  design  in  the  desired  way.  Notationally , 
the  problem  is  how  to  further  represent  it  so  that, the  appropriate 
realization  environment  can  bring  forth  the  actual  realizations. 

In  software,  the  language  in  which  the  design  is  specified  to  the 
detail  part  level  (i.e.,  the  operational  relationship  level)  is, 
by  definition,  not  usable  at  the  higher'  and  intermediate  levels  of 
design.  And,  of  course,  any  truly  general  software  design  process 
m\2st  not  only  be  independent  of  a specific  language  but  even  of  a 
leuiguage  type.^  Therefore,  it  must  be  equally  applicable  to  higher 
level  procedural  and  RPG  languages,  as  well  as  assembly  code  and 
even  binary  machine  instructions.  A multitude  of  possible  realiza- 
tion environments  must  be  acceptable  for  the  same  reasons.  And, 
because  a design  needs  to  present  all  the  information  needed  for 
a realization  environment  to  bring  for-th  a realization,  the  notation 
of  sof-tware.  design  must  be  able  to  link  the  "vagueness"  of  higher 
level  design  statements  with  the  ultimate  detail  of  the  operational 
relationships  of  the  design. 

The  process  of  selecting  structures  to  bring  forth  desired  overall 
(functional)  properties  needs  to  be  examined  in  some  detail  at  this 
point  becuase  of  its  rather  considerable  implications  on  the  cost 
and  time  of  design  and  realization.  To  begin  with,  the  idea  of 
the  "key  structure"  of  a design  needs  to  be  introduced.  The  "key 
structure"  idea  relates  to  the  basic  functional  properties  desired 
of  the  design,  and  in  fact  the  basic  key  structure  selected  conditions 
the  entire  remaining  design  and  therefore  the  design  process.  It 
also  limits  the  feasible  limits  of  desi^  modifications  and  enhance- 
ments, both  during  the  original  design  process  and  after  realization. 

* ramson  for  this  comas  from  conplation  of  cha  "fora  fallows  function"  dictum. 
The  conplata  stmtamant  should  be:  "form  follows  function,  function  follo*rs  form, 
and  tha  unitarsa  of  posslhla  dasigns  is  dafinad  by  tha  sat  of  known  stzucturas 
and  thair  propartias."  A spacific  languaga  thus  savaraly  limits  tha  sats  of 
possibla  daslgns  by  its  possibla  oparational  ralationships . 


A key  structure  is  that  choice  of  form  to  achieve  the  most  basic 
functional  property  of  the  design  realization.  It  is  perhaps  most 
easily  explained  by  several  examples.  Returning  to  DNA,  and  there- 
fore to  the  set  of  all  species  (both  living  and  extinct) , several 
choices  exist  for  providing  a permanent  shape.  A partial  list 
includes  shells,  ekoskeletons , and  a vertebrate  skeleton.  Given  a 
selection  of  one  of  these  "key  structures,"  the  remainder  of  the 
"design"  off  the  species  becomes  limited  to  a rather  few  basic  overall 
forma.  Molluaka  all  look  like  molluaks;  lobsters,  crayfish,  and 
craibs  are  basically  similar  in  shape,  and  vertebrates  are  instantly 
recognizaible  as  vertebrates.  The  key  structure  is  a basic  design 
characteristic . 

A second  example  from  the  set  of  entities  designed  by  humans  rather 
than  evolution  is  perhaps  more  illuminating  because  one  can  visualize 
the  thinking  process  that  goes  into  the  actual  design  process.  Con- 
sider the  basic  functional  requirement  for  a device  to  travel  from 
one  physical  location  to  some  other  location.  The  possible  key 
structure  choices  here  reflect  things  like  wheeled  vehicles  (bicycles , 
automobiles,  trains,  etc.),  flying  things  (balloons,  gliders,  powered 
aircraft),  rockets,  ground  effect  devices,  ships,  etc.  Each  key 
structure  will  accomplish  the  basic  functional  goal;  i.e.,  has  proper- 
ties which  interact  with  the  presiamed  environment  of  the  design  utili- 
zation in  a way  that  permits  travel  from  one  location  to  another. 

Each  key  structure,  when  selected,  essentially  defines  the  remainder 
of  the  design  problem.  And,  all  possible  designs  built  around  a 
given  key  strucfure  can  only  have  other  properties  within  limited 
ranges.  For  exeunple,  the  speed  of  a hot  air  balloon  can  never  approx- 
imate the  speed  of  a rocket.  . Nor  can  the  cargo  czLrrying  ability  of 
a rocket  approximate  that  of  a ship  - at  least. not  with  currently 
known  key  structures  to  build  a rocket  around.  So,  the  key  structures 
for  a given  design  can  only  be  selected  when  all  of  the  basic  func- 
tional requirements  are  properly  defined. 
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Aji  understanding  of  the  role  of  key  structures  and  of  the  selection 
of  a key  structure  from  a knowledge  of  functional  requirements  is 
absolutely  essential  if  changing  functional  requirements  are  to  be 
effectively  managed.  The  proper  understanding  permits  one  (a)  to 
know  how  to  specify  functional  requirements,  (b)  which  types  of 
functional  requirements  reflect  fundamental  design  considerations, 

(c)  what  the  impact  of  chamged  or  additional  functional  requirements 
will  be  on  the  crirrent  design  and  the  current  state  of  the  design 
process,  and  (d)  what  the  implications  are  of  each  functional  require- 
ment in  terms  of  structural  (design)  complexj.ty  and  the  cost  and 

time  reqxiired  to  design  and  realize  the  desired  end  item.  Among 
additional  benefits,  the  ability  to  partition  a design  into  sub- 
structures with  known  interfaces  is  enhanced  considerably,  simpli- 
fying the  design  management  task. 

Again,  an  example  is  probably  in  order,  especially  relative  to  point 

(d)  above.  Assiame  a collection  of  fvinctional  software  specifications, 
which  includes  one  requirement  which  is  relatively  unimportant  to 

the  basic  purpose  for  which  the  software  is  to  be  built.  Assume 
this  "unimportant"  specification  is  that  on-line  updates  to  a data 
base  can  be  made,  even  though  it  is  almost  always  assumed  that  the 
data  base  will  be  updated  "off-line."  This  "unimportant  specifica- 
tion" will  cause  the  "key  structure"  for  the  data  base  to  be  a sig- 
nificantly more  complex  structure  than  needed  if  only  off-line  up- 
dating is  permitted.  This  added  complexity  will  cause  a step  func- 
tion increxnent  in  the  costa  and  time  to  design,  and  to  the  opera- 
tional "cost"  in  the  sense  of  consumption  of  computing  resources. 

Thus,  it  is  essential  to  review  functional  specifications  against 
their  impact  in  requiring  more  costly  key  software  structures. 

Key  structure  considerations  obviously  are  not  limited  to  the 
original  design  process,  but  must  be  applied  to  the  question  of 
enhancements  over  time.  A desired  functional  requirement  applied 
to  a design  whose  key  structure  cannot  provide  the  properties  cam 
only  result  in  essentially  a new  design.  If  it  is  "kludged"  in 
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rather  than  redesigned,  the  complexity  of  the  resultant  code  can 
easily  lead  to  an  unmaintainable  system. 

The  key  structure  concept  can  be  applied  to  any  level  of  design, 
and  is  not  limited  to  only  the  overall  dfcsign.  Therefore  a hier- 
archy of  key  structure  design  c.hoices  appears  during  the  actual 
design  process.  The  major  practical  result  of  amy  key  structure 
selection  is  to  significantly  constrain  the  design  choices  that 
are  possible  after  the  selection.  As  a result,  as  more  and  more  of 
these  decisions  are  made  the  possible  choices  of  code  become  more 
and  more  limited.  At  the  ultimate  "detail  part"  or  code  level, 
little  choice  really  remains  and  so  the  "coding  activity"  is 
normally  treated  as  a very  low  level  activity.  This  is  true  only 
if  no  key  structure  decisions  are  being  made  at  coding  time.  In 
the  absence  of  an  explicit  understanding  of  key  structures,  coding 
often  does  involve  some  key  structure  decisions,  and  when  this  is 
so  the  res\ilt  is  usually  added  complexity  and  obscure  code  for 
maintenance  purposes.  At  the  least,  it  is  a decision  that  needs 
to  be  carefully  made  in  the  full  context  of  the  functional  require- 
ments on  the  structure  selected,  not  from  the  limited  view  of  the 
coder  at  the  time  of  selection. 


i. 

i. 
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I have  chosen  the  term  "design  plateau"  to  reflect  the  impact  of 
key ' structure  decisions  on  design  complexity  and  the  concomitant 
cost/time  implications.  Visually,  the  complexity  of  a design  remains 
relatively  constant  for  a given  set  of  higher  level  key  structures, 
but  shows  a step  function  behavior  between  key  structures . As  an 
example,  think  of  the  design  complexity  of  a child's  wagon,  a 
bicycle,  an  automobile,  and  a train.  The  design  costs  and  time 
needed  are  clearly  about  the  same  for  all  possible  wagons,  for  all 
possible  bicycles,  etc.  But  each  "design  family”  has  a signiflcemtly 
different  level  of  cost/time  associated  with  it. 
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A brief  conment  on  the  implication  of  key  structure  concepts  on 
the  "top-down"  versus  "bottom-up"  design  argiunents  is  appropriate 
at  this. point.  If  a key  structure  is  well  known  to  the  designer 
in  terms  of  its  structural  requirements  and  functional  behavior, 
then  a pure  "top-down"  approach  is  possible.  For  example,  if  one 
is  dealing  with  a purely  sequential  file  structure,  say  for  a log 
tape,  then  no  surprises  should  be  expected  during  the  design  process. 
The  designer  can  see  ahead  sufficiently  to  the  structure  to  use  an 
hierarchic  design  approach  and  not  get  in  trouble  at  the  more 
detailed  levels. 

On  the  other  hamd,  if  the  key  structure  properties  are  complex  and 
especially  if  they  are  complex  and  the  designer  is  not  broadly  ex-  ' 
perienced  in  their  use,  then  a strong  need  exists  to  carefully 
detail  the  structure  before  design.  Detailed  coding  may  also  be 
needed  to  explore  the  properties  of  the  structure,  and  to  establish 
how  the  desired  functional  properties  are  to  be  provided.  In  this 
instance,  the  "bottom-up"  design  approach  appears  to  be  needed. 
However,  note  that  what  is  really  happening  is  that  the  designer  is 
exploring  the  implications  of  a high  level  design  decision,  so  that 
the  constraints  generated  by  it  are  well  known  and  can  be  provided 
for  at  more  detailed  design  levels.  This  is  the  so\irce  of  confusion 
between  the  two  approaches,  and  an  understanding  of  it  clearly 
identifies  when  detailed  code-level  analyses  are  required. 

The  problem  being  addressed  by  the  Software  Life  Cycle  Management 
Workshop  is  how  to  manage  the  design  process  over  the  life  of  a 
software  product.  Implicit  in  the  discussions  is  a view  of  the 
"design  process"  currently  in  use,  and  the  simple  fact  is  that  the 
costs  and  time  are  as  much  a function  of  the  design  methodology  as 
the  design  itself.  This  comes  from  the  absence  of  an  explicit  under- 
standing of  how  to  partition  designs  (i.e.,  assign  levels  of  key 
structure) , how  to  notationally  describe  a design  at  high  and  inter- 
mediate levels  in  a directly  subs true tur able  form,  and  the  relation- 
ship between  functional  requirements  2md  structural  complexity  of 
a design. 
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In  particular,  management  control  and  resource  allocation  over  a 
software  life  cycle  are  directly^ related  to  a recognition  of  these 
problems.  A satisfactory  resolution  of  these  problems  will  never 
be  obtained  in  the  absence  of  an  explicity  and  formal  understanding 
of  the  design  process,  and  this  in  turn  requires  a similar  under- 
standing of  the  properties  of  a "design"  in  the  sense  at  least 
similar  to  that  considered  here.  The  practical  fact  of  the  world 
today,  however,  is  that  these  design  concepts  are  not  written  up 
rigorously,  nor  have  they  been  demonstrated  in  a sufficient  number 
of  instances  to  be  used  to  teach  the  techniques.  A "realization 
environment"  does  not  exist  yet  for  them.  On  the  other  hand,  the 
Army  has  many  projects  currently  underway  and  proposed  for  implemen- 
tation in  the  coming  year  or  so,  and  these  must  be  done  with  current 
technical  and  managerial  skills.  So,  it  is  clear  that  the  workshop 
contributions  are  needed  for  the  short  term  time  frame  (up  to  the 
next  3-5  years) . But  any  basic  set  of  solutions  must,  I believe, 
be  based  on  a software  engineering  design  methodology  which  is  fully 
founded  on  a rigorous  theory  of  designs. 

Since  the  avowed  basic  purpose  of  this  workshop  is  to  define  research 
projects  to  resolve  the  current  life  cycle  management  problems,  I 
strongly  recommend  this  research  be  funded  and  begun.  In  terms  of 
money  and  effort,  it  should  be  kept  small,  because  it  is  not  amenable 
to  "team  research."  The  Army  should,  I suggest,  pick  a few  people 
with  a d9mon3tvat^d  ability  to  develop  ideas  of  this  scope  and  with 
practical  experience , and  "bet  on  them. " 


186 


BEYOND  THE  FOUR  STAGES  : 
WHAT  NEXT  ? 


AUGUST  22,  1977 


David  6.  Robinson 


D.P.  MANAGEMENT  CORPORATION 
One  Militia  Drive 
Lexington,  Mass.  02173 
617-862-8820 


INTRODUCTION 


I 


The  Stages  Hypothesis  of  DP  growth developed  by  Richard  L.  Nolan  of 
D.P.  Management  Corporation,  has  proven  to  be  a useful  and  reliable  means  of 
assessing  the  status  of  an  organization's  information  processing  activities. 
Originally  based  on  three  Harvard  Business  School  research  studies  in  the 
early  1970's,  the  Stages  Hypothesis  has  since  been  verified  in  consulting 
studies  conducted  in  over  40  organizations. 

Since  the  original  statement  of  the  Stages  Hypothesis  in  1973,  we  have 
noted  organizations  whose  DP  growth  and  development  has  tracked  well  with 
Stages  I -I II,  but  which  are  diverting  from  the  predicted  movement  to  Stage 
IV.  These  developments  have  caused  us  to  think  further  about  the  stages  pro- 
gression and  to  postulate  the  DP  growth  characteristics  beyond  the  fourth 
stage.  This  paper  presents  some  of  our  thinking  in  this  area.  It  is  organ- 
ized into  four  sections: 

• Overview  of  the  Stages  Hypothesis 

• Recent  Developments:  Thoughts  About  the  Fifth  Stage  and 
Stages  V and  VI. 

• One  Approach:  Maintenance  as  the  "Driver" 

• "Repetitive  Stages":  An  Alternative  Hypothesis 
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See  Nolan,  Richard  L, , "Managing  the  Computer  Resource:  A Stage  Hypothesis. 
Communications  of  the  ACM.  Vol.  16,  No.  7 (July,  1973),  pp.  399-405. 
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OVERVIEW  OF  THE  STAGE  HYPOTHESIS 


The  Stage  Hypothesis  was  developed  by  Holan  out  of  an  examination  of  the 
patterns  of  growth  in  data  processing  expenditures  in  three  organizations. 

When  total  data  processing  expenditures  were  plotted  against  time,  the  result- 
ing curve  was  "S-shaped."  Nolan  theorized  that  the  S-shaped  curve  was  a mani- 
festation of  underlying  forces,  and  looked  more  closely  at  the  characteristics 
of  the  study  organizations  at  various  points  in  time.  This  led  to  the  hypoth- 
esis that  the  S-curve  articulates  the  progression  of  an  organization  through 
four  identifiable  stages  of  OP  growth,  each  stage  representing  a new  phase  in 
the  organization's  overall  "learning"  about  information  processing  (Exhibit  1). 


I 1 

I STAGES  Of  OP  growth  EXHIBIT  I ■ 


The  four  stages  were  defined  as  follows: 


• Stage  I:  Initiation 

During  Stage  I the  computer  is  introduced  into  the  organization  and 
basic  computer  technology  is  assimilated.  After  the  initial  expendi- 
ture for  DP  hardware  and  personnel,  growth  remains  slow  and  steady. 

• Stage  II:  Contagion 

Stage  II  is  marked  by  a dramatic  upswing  in  expenditures  and  sustained 
rapid  DP  growth.  New  applications  proliferate  in  all  functional  areas 
as  the  organization  applies  the  technology  initiated  in  Stage  I. 

• Stage  III:  Consolidation 

The  rapid  growth  and  uncoordinated  proliferation  of  Stage  II  eventually 
results  in  growing  concern  on  the  part  of  the  organization  and  its 
management  about  the  effectiveness  and  efficiency  of  its  DP  activities. 
This  is  especially  true  since  the  rapid  growth  of  Stage  II  is  usually 
accompanied  by  weak  or  non-existent  formal  DP  controls.  As  a result 
of  the  reaction  to  Stage  II,  DP  growth  is  curtailed  and  formal  controls 
are  introduced.  In  some  cases.  Stage  III  controls  are  excessive  and 
result  in  a regression  in  DP  capabilities. 

• Stage  IV:  Integration 

Over  time,  the  forces  of  control  and  growth  reach  a balance,  and  steady, 
managed  growth  can  be  pursued,  marking  the  entry  into  Stage  IV. 

Because  of  the  underlying  forces  which  drive  the  overall  stage  progression 
are  exceedingly  complex,  the  overall  D?  activity  of  the  organization  was  bro- 
ken down  into  four  principal  components,  each  of  which  has  its  own  unique  stage 
characteristics: 

• The  Applications  Portfolio  of  information  systems,  which  is  the  cumu- 
lative  end  product  of  the  overall  DP  effort. 

• The  DP  Organization,  which  contains  the  technical  resources  (hardware, 
software,  special ized  techniques),  personnel  resources,  and  organiza- 
tion structures  which  can  be  applied  to  DP  efforts. 

• DP  Management  Planning  and  Control  Techniques 

• The  User  of  data  processing  systems. 

Exhibit  2 summarizes  the  characteristics  of  each  of  the  four  components  in  each 
stage. 

In  summary,  the  Stage  Hypothesis  presents  a framework  for  understanding 
the  evolution  of  DP  within  an  organization,  and  the  way  in  which  organizations 


learn  about  information  processing. 
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19T5  by  Richard  L.  Nolan.  Reorinced  by  parr*isslQr. 


Further  Work  with  the  Stages 

Since  the  development  of  the  Stage  Hypothesis  in  1973,  our  work  at  O.P. 
Management  Corporation  has  given  us  an  opportunity  to  validate  and  further  de- 
velop the  Hypothesis  in  over  40  consulting  applications  to  date  in  the  public 
and  private  sectors.  We  have  found  that  the  DP  growth  pattern  of  these  organi- 
zations has  tracked  exceedingly  well  with  the  Stages  theory. 

In  our  work,  we  use  the  Stages  theory  as  the  basis  for  a DP  assessment 
technique  we  call  a "Stage  Audit."  The  Stage  attributes  shown  on  Exhibit  2 
sunmarize  "benchmarks"  against  which  the  particular  characteristics  of  a given 
organization  can  be  measured.  This  gives  us  a broad  and  wel 1 -integrated  frame- 
work against  which  to  view  specific  observed  problems  and  formulate  solutions. 
For  example,  one  of  our  clients  had  undertaken  a highly  ambitious  on-line  data 
base  program  in  an  organization  with  inexperienced  DP  personnel  and  an  almost 
total  lack  of  DP  controls  and  management  techniques.  This  use  of  advanced 
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stage  III-IV  technology  in  an  organization  with  Stage  I-II  skills  and  manage-  jl 

■ ment  techniques  resulted  in  a painful,  overlong  implementation  which  seriously 

damaged  user  confidence.  The  Stages  analysis  showed  the  need  to  advance  DP  ■■ 

personnel  skills  and  management  techniques  to  Stage  III-IV  levels  coordinated  *' 

with  the  technology  in  use,  before  advancing  further  into  significant  new  ap-  t 

plication  system  projects. 

When  we  examine  the  degree  to  which  these  in-depth  studies  validate  the  'i 

Stages  Hypothesis,  we  find  that  our  results  track  exceedingly  well  with  Stages  | 

I-III.  Stage  IV,  however,  has  always  been  more  difficult  to  deal  with  than  ' 

i the  other  stages,  since  so  few  organizations  have  achieved  the  level  of  DP 

I sophistication,  maturity  and  stability  it  defines.  Recent  developments  in 

I Stage  III  organizations  we  have  observed  have  also  indicated  a divergence 

I from  the  predicted  transition  to  Stage  IV.  It  is  these  developments  and  their 

i.  implications  for  the  original  Stages  Hypothesis  which  we  shall  discuss  next. 


RECENT  DEVELOPMENTS; 

THOUGHTS  ABOUT  THE  FIFTH  STAGE  AND  STAGES  V AND  VI 


Recently,  we  have  noted  organizations  in  Stage  III  which  do  not  continue 
the  stable  growth  pattern  predicted  for  Stage  IV.  Instead,  their  expenditures 
have  begun  to  rise  markedly  again,  as  they  apparently  enter  a new  phase  charac- 
teristic of  the  earlier  Stage  II  (Exhibit  3).  Nolan  first  documented  this 
phenomenon  in  1975,^^^  and  theorized  that  it  was  the  result  of  the  assimilation 
of  major  new  technology.  He  presented  three  newly  emerging  technologies  as 


alternative  explanatory  hypotheses:  data  base  management,  minicomputers,  and 

word  processing.  Data  base  technology  appeared  to  be  the  central  focus  of 

(2) 

this  new  growth  resurgence.' 

From  that  work,  Nolan  has  recently  postulated  an  extension  of  the  original 
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four  stages  to  include  Stages  V and  VI,  as  shown  on  Exhibit  4.'  ' Here,  the 
DP  growth  pattern  originally  experienced  in  the  Stage  I-IV  progression  describes 
the  assimilation  of  computer  technology  into  the  organization;  the  progression 
of  Stages  III-VI  is  similar  in  character,  but  describes  the  assimilation  of 
data  base  technology.  Exhibit  5 postulates  the  specific  characteristics  of 
the  Stage  III-VI  progression. 

In  conjunction  with  the  development  of  Stages  V and  VI,  we  have  attempted 
to  examine  the  forces  which  might  be  responsible  for  this  shift  to  rapid,  data 
base  centered  growth.  The  next  section  outlines  one  approach  we  have  developed 
using  quantitative  data. 


STAGES  or  EVOLUTION  OE  THE  OATA  SESOL-’';  FViCTION 


EXHtaiT  4 
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ASSIMILATION  OF  COMOUTEW  TICHWOtOGY 


stage  I stage  II  stage  III  stage  IV  stage  V STAGE  VI 


•tr  n/S  by  L.  H«»nrlnfed  by  b^rmis^ioo. 


^'Nolan,  Richard  L. , "Restructuring  th-  Data  Processing  Organization  for  Data 
Resource  Management."  IFIPS  1977  Conference  Proceedings. 


ONE  APPROACH:  MAINTENANCE  AS  THE  DRIVER 
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One  of  the  measures  we  attempt  to  gather  and  analyze  in  our  Stage  Audit 
consulting  studies  is  the  ratio  of  new  system  development  to  maintenance  across 
the  Stages.  Our  benchmarks  for  this  measure  are  shown  in  Exhibit  6.  Stage  I 
should  normally  be  almost  entirely  new  development.  In  Stage  II,  maintenance 
begins  to  appear,  but  new  development  still  predominates.  In  Stage  III,  how- 
ever, the  balance  shifts  as  rapid  expansion  is  curtailed  and  the  organization 
concentrates  on  maintaining  the  Stage  II  applications  base  to  meet  changing 
user  requirements. 


• I 

i -AINTE((ANCE-(WtV£‘(  ’ECO  EXHIBIT  6 1 
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Obtaining  hard  data  in  this  area  is  difficult,  since  many  organizations 
do  not  maintain  useful  data  on  the  ratio  of  development  effort  to  maintenance 
effort.  Exhibit  7 presents  the  results  of  20  observations  we  have  been  able 
to  collect  over  the  past  three  years.  Of  these  20,  15  showed  development/ 
maintenance  ratios  which  were  typical  of  the  expected  balances  shown  on  Ex- 
hibit 6,  while  5 were  atypical.  An  average  of  all  20  cases  shows  a rough 
ratio  of  60%  new  development  and  40%  maintenance  in  Stage  II,  shifting  to  30: 
70  in  Stage  III  as  maintenance  predominates. 


STAGE  MAINTE’lAflCE  DATA  EKHISIT  7 


• 20  03StRVAT10;iS,  15  'riPiCAL" 

i 

i 

• STAGE  I i AVERAGE:  AZZ  ;'AiMEf.A;iCE 

i S8Z  DEVELOP.rEiNT 

• STAGE  III  AVERAGE:  67:  V.AIfiTEflAiTCE 

53:  OEVElCP-'-'ENT 

i 

We  view  maintenance  levels  as  an  indicator  of  user  satisfaction  and  sys- 
tem effectiveness.  A given  system,  once  developed,  can  be  maintained  to  retain 
user  satisfaction  and  effectiveness.  But,  as  shown  on  Exhibit  8,  there  is  a 
limit  to  the  amount  of  periodic  maintenance  and  enhancement  which  can  be  sub- 
stituted for  a complete  system  "re-do,"  as  the  effectiveness  of  maintenance 
progressively  declines  over  time  and  its  cost  progressively  increases.  When 
the  maintenance  cost/benefit  ratio  is  no  longer  positive,  a complete  system 
overhaul  is  initiated. 


197 


HAifi't.'ii.'.Ce  IS  AH  [hc:ca:cii 


EIH13IT  8 


i 


/ 


I 


I 

\ 

( 


While  this  movement  for  complete  system  rewrites  occurs  throughout  the 
Stages  progression,  it  tends  to  concentrate  in  late  Stage  III,  as  the  "baby 
boom"  of  Stage  II  systems  mature  at  roughly  the  same  time.  The  high  mainten- 
ance ratios  we  observe  in  Stage  III  (Exhibit  7)  are  indicators  of  this  overall 
tendency.  Thus,  during  Stage  III  it  is  likely  that  a major  overhaul  of  ex- 
isting systems  will  be  indicated.  We  see  this  occurring,  with  data  base  tech- 
nology embraced  as  the  vehicle  for  carrying  it  out.  The  broad  scope  of  the 
data  base  program  pushes  the  organization  into  the  Stage  III-VI  learning  pat- 
tern as  it  assimilates  data  base  technology. 

Nolan's  current  thinking  sees  Stage  VI  as  maturity  in  the  evolution  of 
the  management  of  the  Data  Resource  Function. As  the  author  has  examined 
the  need  to  extend  the  original  Stages  Hypothesis,  he  has  begun  to  consider 
an  alternative  to  Nolan’s  six-stage  hypothesis.  This  alternative  hypothesis 
is  presented  in  the  following  section. 


1 


! 


1 


i 


j 


The  "Repetitive  Stages"  explanation  views  Stages  V and  VI  not  as  just  a 
one-time  adjustment  of  the  hypothesis,  but  rather  a specific  articulation  of 
a more  general  theory.  Exhibit  9 presents  the  author's  idea  of  repetitive 
stages  of  growth,  each  of  which  represents  the  learning  of  a new  information 
technology  by  the  organization.  The  learning  of  computer  technology  thus 
occurs  in  Stages  I-III;  Stage  III  then  becomes  the  base  for  the  learning  of 
data  base  technology  in  Stages  III-V,  and  so  on  into  new  areas  of  informa- 
tion technology  about  which  we  can  only  speculate  today. 


Using  this  explanation  of  currently  observed  events,  the  system  "re-do" 
of  Stage  III  is  a repetitive  phenomenon  which  occurs  at  different  times  for 
different  organizations.  For  example,  while  some  organizations  v/e  observe 
today  are  in  Stage  III,  others  are  still  in  Stage  II  and  will  not  reach  Stage 
III  for  years.  When  an  organization  does  reach  the  point  in  Stage  III  at 
which  it  perceives  the  need  for  a major  systems  overhaul,  it  will  tend  to  as- 
similate the  next  major  increment  of  information  technology  as  the  focus  for 
the  overall  program. 

Today,  data  base  technology  predominates  as  the  vehicle  we  observe  being 
selected.  Yet  the  author  theorizes  that  organizations  which  are  today  entering 
a new  data  base  oriented  growth  period  in  Stages  III-V  will  again  emerge  into 
a period  of  consolidation,  only  to  eventually  perceive  the  need  for  another 
major  "re-do"  in  late  Stage  V.  At  this  point,  a new  information  technology 
would  be  embraced  and  a new  stages  learning  progression  entered.  We  can  only 
speculate  on  the  technology  of  the  future,  although  there  are  indications 
that  word  processing  and  office  automation  may  be  likely  candidates. 

Taken  more  broadly,  the  "Repetitive  Stages"  concept  tracks  well  with  the 
idea  of  an  emerging  data  resource  function, in  which  each  stage  progression 
represents  a new  advancement  in^the  evolution  of  an  information-oriented  manage- 
ment function.  Today's  data  processing  function  focused  on  computers  is  only 
a subset  of  the  broader  Data  Resource  Function  of  the  future,  focused  on  the 
information  resource.  As  with  other  specialized  management  functions  such  as 
Production,  Marketing  and  Finance,  Data  Resource  management  must  be  learned 
over  time.  Nolan  sees  Stage  VI  as  the  stage  of  maturity  in  the  evolution  of 
the  Data  Resource  Function.  The  author's  alternative  hypothesis  postulates 
additional  stages  of  learning  and  technology  assimilation  beyond  Stage  VI. 

In  this  context,  we  can  view  the  history  of  information  management  as  a series 
of  evolutionary  learning  jumps  as  shown  on  Exhibit  10.  Each  is  a significant 
step  in  the  development  of  the  Data  Resource  Function. 


Try 


See  Richard  L.  Nolan,  Managing  the  Data  Resource  Function,  (St.  Paul, 
Minnesota:  West  Publishing  Company,  1974). 
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Our  Next  Steps 

We  have  found  the  Stages  Hypothesis  to  be  a powerful  tool  in  assisting 
organizations  with  the  management  of  their  information  processing  activities. 

We  are  currently  updating  our  methodology  continuously  to  deal  with  the  condi- 
tions encountered  in  the  Stage  III-VI  progression.  Beyond  this,  the  author 
intends  to  continue  to  examine  his  "Repetitive  Stages"  hypothesis  as  additional 

empirical  data  is  gathered  in  our  consulting  work.  J 

1 
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LIFE  CT/CLE  PLAIT^ING  FOR  A LARGP:  MIX  OF  CO>!:iFRCIAL  SYSTEMS 


by 


I.  R.  EllioCw 

Systems  Si  Progransaing  Manager 
Midland  3ank  Limited  (U.K.) 


1. 


Introduction 

Midland  Bank  has  a centralised  Systems  & Programming  Division  with  over 
200  staff,  maintaining  and  developing  commercial  systems  for  a variety 
of  users  within  the  Bank.  Several  Computer  Centres  in  the  U.K.  are 
used.  Due  to  our  large  investment  in  present  systems  we  have  a natural 
interest  in  preserving  them  for  a reasonable  time,  and  planning  their 
renewal.  We  can  claim  from  e:-:perienca  a working  knowledge  of  the 
factors  which  compromise  t.he  life  of  a system,  but  our  introduction  to 
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Professor  M.  M.  Lehman  of  Imperial  College,  London.  Under  his  direction 
the  evolution  of  a particular  system  was  studied.  It  is  the  purpose  of 
this  p^p^r  to  discuss  the  future  of  sll  o«»r  wicli  pevtlcul^^x 


reference  to  the  resources  required  to  sustain  them.  The  manpower 
strength  of  Systems  & Programming  in  recent  years  is  shown  in  Figure  A. 
(all  figures  are  at  the  end  of  the  paper).  It  will  be  seen  the  manpower 
has  grown  steadily  and  while  we  can  excuse  this  in  various  ways,  claiming 
a spate  of  original  and  renewed  systems,  we  would  not  like  to  see  the 
effort  double  again.  The  ageing  of  systems  and  the  phenomena  of  low 
productivity  and  paralysis  leading  to  "re-writes"  has  been  masked  in  the 
past  by  otner  factors  but  will  be  much  more  significant  in  future  with 
the  complexity  of  systems  now  being  supported. 


2.  Approach 

A mixture  of  theory,  or  rather  modelling,  with  observation  of  our  actual 
situation  has  been  used.  As  it  is  inherently  difficult  to  quantify  a 
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subject  wich  so  many  variables,  it  is  useful  to  check  exploratory 
conclusions  against  a feel  for  an  actual  situation.  The  variety  in 
both  scope  and  characteris cics  of  systems  in  the  mix  appears  to  be 
sufficient  for  generalisation  of  our  situation,  but  no  claim  is  made  that  .. 

our  model  can  be  transported  to  another  situation.  Rather  it  is  an  ^ 

approach  or  a way  of  thinking  which  may  be  of  some  use.  Also  our  own 
thinking  on  program  evolution  is  at  such  an  early  stage  that  we  prefer 
to  check  with  others  before  constructing  a too  shaky  or  parochial 
theoretical  edifice. 

The  question  ve  are  asking  is  not  so  much  "how  long  will  systems  last?" 
as  "how  long  do  we  need  to  make  systems  last?".  The  life  cycle  of  a 
system  is  subject  to  a large  degree  of  management  control,  albeit 
drastic  measures  are  needed  in  some  cases.  The  influence  of  those 
responsible  for  development  is  of  course  just  one  of  the  many  environ- 
mental pressures  or  constraints  on  the  evolution  of  systems. 

3.  Definitions 

We  can  for  the  time  being  permit  ourselves  a certain  looseness  about  the 
definition  of  "system"  as  we  shall  address  the  characteristics  of  the 
total  program  source  code  as  the  moat  objective  quantifiable  factor  of 
the  situation.  In  considering  the  evolution  of,  for  example,  a mix  of 
systems  totalling  one  million  lines  of  source  code,  this  may  be  done  at 
the  "atomic"  level  of  individual  code  lines  or  as  (say)  a thousand 
systems  or  sub-systems  of  one  thousand  lines  each.  In  fact,  an 
elementary  model  has  been  constructed  and  then  the  back-to-frent  approach 
has  been  taken  of  considering  which  useful  situations  are  catered  for  by 
the  model. 

One  problem  of  Life  Cycle  theory  is  the  amount  of  introduction  required 
before  any  progression  is  made,  particularly  as  each  stage  of  definition 
leads  to  reflection  on  its  validity  in  practical  terms.  Let  us  first 
make  the  fairly  obvious  distinction  between  age  and  life  times  as  applied 
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CO  source  code.  The  code  is  considered  "bom"  when  it  goes  live 
operationally,  Chen  its  age  may  be  measured  up  to  a point  of  time  such  as 
Che  present.  Its  life  is  Che  time  from  birch  to  "death"  when  Che  code 
or  system  is  discarded.  Considering  a system  as  an  agglomeration  of 
code,  systems  have  a certain  age  since  going  "live"  and  a certain  life 
before  they  are  typically  renewed . We  shall  generally  consider  code  or 
system  renewal . Our  systems  are  part  of  the  organisation  we  support 
and  therefore  every  system  becoming  defunct  must  generally  have  a 
successor  system  (or  sub*system,  etc.)  prepared.  In  fact,  Che  renewal 
of  an  entire  system  is  only  a special  case  of  the  measures  needed  to 
adapt  it  to  its  environment,  buc  one  with  major  planning  and  resource 
implications . 

4.  Program  Census 

To  provide  a base  of  code  and  systems,  a census  was  taken  of  all  live 
code  for  which  Systems  & Programming  was  responsible  on  1 July,  1977. 

The  source  code  was  classified  by  machine,  language,  and  Che  dates  it 
originally  went  live.  For  Che  remainder  of  this  paper  we  consider 
amounts  of  source  code  lines  irrespective  of  langxiage  used.  There 
was  a total  of  1,918,000  source  lines.  The  average  age  of  Che  code 
is  2 years  7 months . 

The  distribution  of  the  code  into  its  years  of  origin  is  shown  in 
Figure  B in  histogram  fora.  It  is  simpler  for  further  extrapolation 
to  present  the  data  by  measuring  time  from  a fixed  date  in  the  past 
rather  than  relative  to  the  date  of  the  census.  It  should  be  noted 
chat ; - 

a.  The  origin  of  the  horizontal  (time)  axis  is  1 July,  1967.  Amounts 

of  code  are  shown  in  yearly  intervals  up  to  1 July,  1977. 

b.  The  ten  years  shown  will  be  referred  to  as  "Year  1"  up  to  "Year  10". 
Thus  we  still  have,  for  example,  50,000  lines  of  code  from  Year  1 
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(prcduccd  batvcer.  1 July,  1957  aud  1 July,  1958).  359,000  lines  of 

new  code  went  live  in  Year  10  (between  1 July,  1976  and  1 July,  1977). 

c.  It  should  be  remembered  chat  this  is  not  a chart  of  Che  code  we  have 
produced  over  the  past  ten  years,  but  that  which  has  survived. 

5.  Renewal  Model 

Suppose  for  the  moment  code  will  cnly  last  for  a certain  time  before  it 
is  renewed,  but  chat  the  total  amount  of  live  code  remains  constant. 

While  it  will  be  necessary  to  inject  further  elements  of  reality  into 
the  situation,  this  is  a useful  basic  "steady  state"  situation.  Whole 
systems  at  a time  may  be  renewed,  or  we  may  be  approximating  the  situation 
where  small  amounts  of  code  are  discarded  and  re-written.  In  this 
situation,  Che  overall  requirements  of  the  organisation  supported  by  the 
several  systems  are  perhaps  changing  but  of  similar  scope.  Indeed  if 
productivity  in  source  lines  was  reasonably  independent  of  language,  Chen 
better  languages  could  enable  Che  organisation  to  accomplish  its  systems 
objectives  with  a constant  amount  of  code.  Having  introduced  this  as  a 

basic  case,  it  is  indeed  tempting  Co  Cry  and  make  it  happen.  However, 

it  is  perhaps  simpler  to  define  Che  model  Chan  Co  be  certain  of  Che 
situations  it  describes. 

a.  An  array  is  used  to  describe  distribution  of  code  from  Year  1 still 
in  existence  at  a given  time.  We  have  generally  initialised  this 
with  Che  amount  of  code  (in  thousands)  from  our  real  situation, 
c.  Using  Che  code  and  probability  arrays,  Che  model  generates  year  by 
year  the  changing  array  of  live  code.  Thus,  starting  in  Year  11, 
for  example,  it  would  calculate  for  each  of  the  previous  years  how 
much  code  should  be  renewed  and  add  these  amounts  into  Year  11  while 
subtracting  them  from  Che  original  years.  Thus,  at  Che  end  of 
Year  11  the  code  has  moved  on  a little  but  its  age  must  now  of  course 
be  measured  relative  to  the  end  of  Year  11.  The  process  is  repeated 
for  each  year  and  ends  after  Year  20. 
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d.  The  new  code  which  has  to  be  generated  each  year  requires  resources 
to  be  allocated  ahead  of  time.  srom  an  amount  of  code  a number  of 
man  years  is  deduced,  then  this  is  spread  and  accumulated  over 
previous  years. 

e.  While  they  are  variables  of  the  model  a rate  of  3,000  source  lines  in 
one  man  year  was  used,  and  then  effort  was  distributed  during  the  two 
years  prior  to  code  going  live.  Approximately  twice  the  effort  is 
applied  in  the  second  compared  with  the  first  year.  life  "coding" 
rate  includes  every  stage  through  feasibility^  study,  analysis  and 
design,  specification,  programming,  system  testing,  etc. 

f . If  this  rate  is  applied'  to  our  existing  base  of  code  it  shows  an 

investment  of  639  man  years  in  our  present  code.  This  compares 

reasonably  with  the  numbers  (Figure  A)  in  Systems  & Programming  when 
\ 

allowance  is  made  for  the  code  that  has  been  discarded  and  the  element 
of  maintenance. 

Simple  Results 

Nothing  more  is  claimed  for  this  method  and  theory  than  it  should  give 
guide-lines . and  for  intelligible  results  to  emerge  the  number  of 
variables  has  to  be  reduced. 

We  decided  that  for  "average  life  m"  we  would  use  an  approximately  normal 
distribution  with  mean  m and  standard  deviation  m/6,  in  order  to  make  it 
a function  of  one  variable.  This  means  that  the  only  significant  part 
of  the  distribution  lies  between  m/2  and  3m/2. 

Using  the  existing  code  base,  we  projected  this  for  ten  years  trying 
average  lives  from  three  to  ten  years . In. each  case  the  array  was 
printed  out  year  by  year  and  finally  the  "effort".  The  effort  shows  the 
nusiber  of  men  in  a year  required  to  produce  the  code  being  renewed, 
allowing  for  the  spread  already  mentioned.  Only  the  effort  from 


Years  11  to  18  is  relevant  since  the  aiodel  stops  (arbitrarily)  after 
Year  20.  The  "peak  effort"  is  the  greatest  number  of  men  required 
between  Years  11  and  18,  while  "average"  is  the  average  over  these  years. 
These  variables  are  plotted  in  Figure  C against  average  life  of  code. 

Looking  at  Figure  C we  must  first  remember  this  shows  only  one  component 
of  Systems  & Programming  manpower,  devoted  to  the  renewal  of  code.  It 
ignores  for  the  time  being  ocher  aspects  of  maintenance,  and  the  development 
of  original  systems  which  expand  Che  total  amount  of  code.  Nevertheless 
it  shows  quite  clearly  that  if  systems  last  only  three  years,  a force 
equivalent  to  the  entire  strength  of  Systems  & Programming  would  be 
devoted  Co  constant  re-writes,  an  obviously  untenable  situation. 

Looking  at  longer  life  times,  Che  peak  requirement  does  not  decline 
against  life  quite  as  rapidly  as  Che  average  because  there  is  a greater 
spread  in  Che  life  distribution  as  average  life  increases. 

One  must  consider  the  deployment  of  other  resources  in  Systems  & 

Prograsoming  before  evaluating  the  desirable  average  life  of  code,  but  my 
own  feeling  is  that  we  must  have  an  average  life  of  seven  years  at  Che 
very  least,  preferably  eight  or  nine,  and  ideally  ten  years. 

Expansion  of  Code  Base 

The  code  base  used  so  far  contains  nearly  two  million  lines  originating 
over  Che  ten  years  up  to  1 July,  1977.  Its  renewal  was  considered  while 
Che  total  remained  constant,  but  "new"  code  can  be  injected  into  Che 
model  year  by  year.  In  fact,  the  array  input  to  Che  model  of  which  the 
ten  years  up  to  "now"  consists  of  existing  code  is  extended  by 
initialising  it  from  Years  11  Co  20  (in  this  case)  with  amounts  of  code 
giving  a net  increase  in  the  total. 

We  modelled  the  situation  where  100,000  extra  lines  of  code  are  added 
each  year  Co  the  code  total.  Thus,  over  a ten  year  period  one  million 
lines  of  code  are  added  bringing  the  total  base  of  code  to  nearly  three 
million  lines. 


It  should  be  emphasised  that  according  to  the  average  code  life  being 
modelled,  the  old  and  new  code  can  be  renewed  more  than  once  during  the 
period  being  modelled. 

As  an  example  after  the  passage  of  ten  years,  i.e.  at  the  end  of  Year  20, 
the  code  base  is  distributed  as  follows  between  Years  10  and  20  if  the 
average  life  is  eight  years: 

11,  24.  73,  165,  267,  355,  426,  451,  422,  373,  351 
e.g.  at  the  end  of  Year  20,.  73,000  lines  remain  which  were 
written  in  Year  12. 

The  resource  implications  of  this  will  be  discussed  in  the  next  section. 
Maintenance 

So  far  we  have  used  the  model  to  evaluate  resources  for  what  may  be 
called  the  dynamic  aspects  of  development  - the  renewal  or  regeneration 
of  existing  applications  and  the  addition  of  further  scope  to  systems, 
increasing  the  total  amount  of  code.  Before  looking  too  closely  at 
vhat  this  really  maens , let  us  have  a look  at  the  supposedly  static 
aspects  of  maintenance.  Of  course,  there  is  in  practice  every  gradation 
between  cade  with  a low  rate  of  change  and  complete  renewal.  The 
dynamic  aspect  of  changing  and  renewing  systems  is  taking  place  against 
a background  of  maintenance  support.  Obviously  it  costs  more  effort  to 
modify  10  lines  of  code  out  of  100,000  than  out  of  (say)  1,000  lines. 

The  "background"  of  code  is  always  there  because  a system  must  remain 
operational  until  its  successor  takes  over.  Thus,  the  total  amount  of 
code  that  is  live,  and  in  practice  expanding,  requires  at  least  a 
proportional  effort  to  be  added  to  the  "dynamic"  aspect  already  modelled. 

What  is  the  inevitable  background  effort  on  which  to  superimpose  the 
"dynamic"  aspects  of  system  evolution? 


To  discuss  this  question  let  oie  first  recklissly  propound  two  "rules  of 
thumb  for  estimating  Che  manpower  required  to  maintain  systems: 

a.  Every  20,000  lines  of  existing  code  requires  a person  to  support  it. 

The  number  of  systems  staff  required  to  maintain  a system  is  one 
quarter  of  the  number  deployed  in  its  final  year  of  development. 

On  this  basis  the  equivalent  of  96  of  my  staff  are  looking  after  the 
1,918,000  lines  of  code  and  another  50  will  be  needed  after  the  amount 
of  code  has  grown  by  another  million. 

Now,  firstly  Che  suintenance  effort  is  not  really  linear  with  systems 
size,  and  secondly  it  depends  on  the  dynamic  activity.  However,  the 
linear  rule  has  more  validity  for  a mix  of  systems  chan  an  individual 
system.  Obviously  in  practice  our  systems  themselves  interact,  but 
up  to  a point  it  is  surely  reasonable  to  say  chat  three  million  lines 
of  code  from  various  systems  require  half  as  much  again  background 
maintenance  as  two  million  lines. 

Let  us  now  summarise  the  manpower  given  by  Che  model  for  ageing  Che 
original  code  with  an  average  life  of  eight  years,  adding  a million 
lines  of  expansion  over  ten  years,  and  calculating  background 
maintenance  as  Che  total  expands. 


Year 

11 

12 

13 

14 

15 

16 

17 

18 

19  20 

Dynamic 

70 

84 

101 

120 

138 

146 

140 

128 

Static 
(year  end) 

101 

106 

111 

116 

1 71 

176 

131 

136 

....... 

Total  people 

171 

190 

212 

236 

259 

272 

271 

264 

This  shows  rather  less  people  chan  we  now  have  in  practice  (at  the  outset 
of  Year  11).  Further  refinement  would  require  a full  discussion  of  the 
actual  composition  of  staff  (managers,  support,  etc.).  It  would  also 
require  a greater  elaboration  of  any  special  factors  in  our  present 
situation,  not  that  there  is  anything  especially  peculiar. 
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ReCuraiag  Co  Che  quescion  o€  background  aainCenance,  when  ic  is  said 
ChaC  one  person  is  required  for  every  exiscing  20,000  lines  of  code, 

Chis  is  Co  supporc  a certain  amounC  of  change  which  has  been  resourced 
under  Che  "dynamic"  cacegory.  The  formula  can  hardly  be  called  accurace 
when  a syscem  which  could  quite  literally  be  frozen  would  require  no 
support.  Put  another  way,  we  may  say  the  linear  formula  for  background 
maintenance  includes  a certain  amount  of  "free"  supporc  and  change  of 
Che  syscem.  Now  in  fact,  as  Che  amount  of  dynamic  change  increases 
against  larger  and  larger  static  amounts  of  system,  the  interaction 
becomes  much  greater  and  we  can  no  longer  separate  the  two  factors  in 
Che  above  Cable.  You  would  need  to  add  a third  line,  a sort  of 
"increasing  complexity"  effect  to  our  resources. 

This  quescion  of  Che  formula  for  resourcing  races  of  change  of  exiscing 
systems  of  various  sizes  opens  up  a subject  in  itself.  Having  scratched 
Che  surface,  it  is  time  Co  conclude  with  a few  general  remarks. 

9 . Conclusions 

It  is  all  very  well  to  consider  how  chose  responsible  for  development 
may  contain  Che  situation  within  assigned  resources,  but  the  ocher 
factors  at  work  make  it  difficult  to  achieve. 

In  Che  future  we  are  working  with  a combination  of  rising  user 
expectations  and  rapidly  changing  technology.  If  "Life  Cycle  Manage' 
ment"  imposes  constraints,  may  it  not  be  seen  as  yet  another  reason 
why  the  systems  people  cannot  do  exactly  what  the  user  wants?  While 
many  people  recognise  intuitively  that  uncontrolled  change  compromises 
a syscem,  the  formal  subject  of  evolution  may  appear  so  esoteric  as  to 
be  merely  the  next  sec  of  excuses. 

Were  it  not  obvious  for  ocher  reasons,  it  can  also  be  seen  ChaC  the 
exploitation  of  new  technology  must  be  a slow  process  according  to  the 
•ocC  vZ  Li)«ux,y  w«  have  discussed.  inis  is  really  based  on  our 
experience  of  supporting  systems  in  the  past,  and  of  course  some 
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technological  developments  (further  distribution,  etc.)  may  enable 
organisations  co  break  out  from  other  restrictions  apparently  beginning 
to  stall  their  progress.  However,  systems  are  systems  and  the  effects 
cannot  be  escaped  in  the  end. 

A particular  problem  of  present  large  systems  is  preserving  continuity 
of  operation  through  the  renewal  process.  A form  of  metamorphosis 
must  be  achieved. 

Finally,  should  we  expect  users  to  plan  their  requirements  for  a more 
specific  time  span?  Should  they  anticipate  the  areas  of  flexibility 
they  require  over  a desirable  time  span?  Certainly  practical  planning 
is  ineffective  without  the  endorsement  of  the  user. 
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tROJECI  LIFE  CYCLE  MCDELLIXG; 

BACKGROUND  AND  APPLICATION  OF  THE  LIFE  CYCLE  CURVES 
PETER  V.  NORDEN,  Ph.D..  P.E. 

IBM  CORPOR-ATION  AND  COLUMBIA  UNIVERSITY 


The  basic  concepio  and  assumptions  of  a multi-stage  model  of  resource  cycles 
in  Development  Projects  are  reviewed.  A few  observations  based  on  experience 
With  this  model  are  presented.  These  are:  Reasonable  stability  of  the  cycles, 
early  patterns  in  projects  that  led  to  success,  and  patterns  observed  when 
projects  changed  scope  in  midstream.  A case  study  illustrating  use  of  the  Life 
Cycle  Curves  on  a hardware  development  project  is  presented,  and  an  .Appendix 
showing  how  to  compute  the  curves  from  simple  Accounting  data  is  given.  The 
effects  of  changes  in  project  environments,  such  as  interactive  programming  and 
debugging,  and  other  technological  advances  are  discussed,  and  the  ''single-cycle' 
possibility  of  software  development  projects  is  noted. 
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PROJECT  LIFE  CYCLE  MODELLING: 

BACKGROUND  AND  APPLICATION  OF  THE  LIFE  CYCLE  CURVES 


INTRODUCTION 

This  paper  consists  of  three  parts : 

1.  Background  ajid  Principles  of  the  Life  Cycle  Curves 

2.  Application  of  the  Life  Cycle  Model  on  a Complex 
Development  Project:  A Case  History. 

3.  Appendices: 

A)  A Theoretical  Model  Explaining  The  Curves 

B)  A 'HOW  - TO'  Manual  for  computing  the  curves. 
(This  has  tabular  appendices  of  its  own) . 


1 


I? 


i 

I 

1 


i 

The  intent  of  this  structure  is  to  present,  in  one  place,  a 
convenient  compilation  of  enough  material  from  earlier  documents 
to  allow  the  reader  to: 

o Become  acquainted  with  the  use  of  the  curves  in  practice 
o 'Walk  through'  a case  study 

o Become  acquainted  with  the  underlying  theory 
o Obtain  a small  reference  manual  for  some 
do-it-yourself  curve  fitting. 

In  addition,  comments  on  more  recent  experience,  particularly  with 
large  software  development  projects,  are  provided.  At  the  risk  of 
some  slight  redundancy,  the  two  sections  and  the  appendices  can  be 
treated  as  stand-alone  documents,  and  are  paginated  accordingly. 
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BACKGRCUIID  PRINCIPLES  OF  THE  CURVES 

The  Life  Cycle  curves,  more  recently  also  called  'Rayleigh' 
curves,  were  discovered  in  the  course  of  a study  I conducted 
between  1956  and  1964  at  the  IBM  Development  Laboratories  in 
Poughkeepsie,  N.Y.  The  objective  of  the  studies  was  to  devise 
improved  methods  for  estimating  and  managing  large  (hardware) 
development  projects.  The  results  were  so  interesting  that  we 
were  able  to  persuade  SCARDE  (the  Study  Committee  for  the  Analysis 
of  Research,  Development,  and  Engineering)  of  the  R&D  College  of 
The  Institute  of  Management  Sciences,  to  allow  me  to  gather 
project-pat-cern  data  from  a large  number  of  other  companies.  The 
data  were  collected,  using  a blind-coding  scheme  administered  by 
Professor  A.K.  Rubenstein  of  Northwestern  University,  which 
assured  confidentiality  of  proprietary  information.  Cycles  of  the 
Rayleigh  form  were  found  in  all  these,  again  predominantly  hard- 
ware, projects,  and  the  analysis  of  the  combined  IBM  and  SCARDE 
data  was  published  as  my  doctoral  dissertation,  and  also  as 
Reference  (5)  . 


The  explanation  of  the  underlying  theory'  is  found  in  Appendix  A. 
It  uses  a probabilistic  model  from  reliability  theory,  originally 
due  to  Professor  Benjamin  Epstein,  and  describes  the  problem- 
solving behavior  of  scientists,  engineers,  and  other  technical 
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wich  probability*  density  functions  to  the  time  series  form  in 


which  manpower  accounting  data  is  conventionally  kept,  and  in 

which  it  is  displayed  in  'life  cycle'  patterns.  These  density 

functions  belong  to  the  family  of  Weibull  Distributions , and  v;e  j 

communicated  cheir  applicability  to  development  management,  around 

1959,  CO  A.  Pietrasanta,  then'  with  IBM's  software  development 

effort  on  t.he  S/360  machine  series.  He,  with  P.  Schreiber, 

early-on  demonstrated  their  applicability  to  software  projects. 

It  is  as  a result  of  the  above  history,  that  we  referred  to  the 

life  cycle  curves  as  'Weibull'  rather  than  'Rayleigh'  curves.  ••  t 

The  curves  were  subsequently  used  with  good  results  in  several  IBM 
locations,  and  in  other  organizations.  Private  communications 

i- 

informed  us  that  among  others,  the  then  ESSO  R&E  organization 

'I 

found  the  cycle  pattern  to  hold;  and  DOD  used  these  findings  to 
institute  a procurement  practice  called  ' Program  Definition 
Phase'  contracting.  IBM,  to  this  day,  has  retained  a development 
protocol  called  'Phase  Reviews',  which  allows  projects  to  be 
reviewed  at  each  cycle  completion  point,  and  authorization  to 

i 

proceed,  as  well  as  funding,  to  be  given  by  stages.  Elsewhere  in 
this  collection  of  papers  you  will  find,  also,  reports  on  the 
massive  work  and  significant  findings  of  Col.  Lawrence  H.  Putnam, 
who  applied  the  curves,  and  refined  their  analysis,  at  USACSC. 

The  principle  of  the  curves  is  as  follows:  Research  has  indicated  I 

that  there  are  regular  patterns  of  manpower  build-up  and  phase-out 
in  complex  projects.  These  patterns  are  made  up  of  a small  number 

t ! 
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f 

of  successive  phases,  or  cycles  cf  work,  throughout  the  life  of 
the  project.  The  cycles  do  not  depend  on  the  nature  or  work-con- 
tent of  the  project,  but  relate  to  the  way  technical  people  tackle 
ccr.plex  development  problems.  Each  cycle  can  be  described  by  the 
comparatively  simple  equation  shown  in  the  'APPLICATION....' 
section,  belcw,  and  yields  a curve  relati.ng  ma.npower  used  each 
month  to  elapsed  calendar  tine  in  the  cycle  The  single  parameter 
(a)  of  the  formula,  governing  the  shape  of  the  curves,  can  be 
thought  of  as  a measure  of  'crashiness'  of  the  project:  sharply 
peaked  manpower  duildups  correspond  to  crash  projects,  while 
shallower  curves  are  associated  with  more  stretched-out  programs. 

Now,  how  do  these  cycles  occur,  and  what  do  they  mean?  The  best 
explanation  is  that  engineering  groups  seem  to  work  in  definite 
surges  of  effort,  and  that  each  surge  is  associated  with  a parti- 
cular purpose  for  doing  the  job. 

In  hardware  development,  the  succession  of  purposes  with  which  we 
work  on  projects  generally  is:  planning,  designing,  building  and 
testing  a prototype,  engineering  activities  associated  with 
release  of  the  product  to  a plant,  occasional  redesign,  and  a 
small  number  of  cycles  for  product  support  and  cost  reduction.  | 

Figure  1 shows  this  sequence  schematically.  Sometimes  it  is  | 

preceded  by  an  exploratory  or  'concepts'  cycle.  In  principle, 
this  sequence  is  very  widespread:  Table  1 lists  the  cycle  nomenc- 
lature established  by  several  sources.  The  'GUIDE'  column  refers 
to  the  computer  users'  group,  GUIDE  International  Inc.,  whose 
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TYPICAL  Vt'ORK  PHASES  (CYCLES)  AND  ACTIVITIES 
ELECTRO- MECH.A  NT  CAL  DEVELOPMENT 
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CYCLE  (PHASE)  NOMENCLATURE 


Installation  Guidelines  and  Standards  Group  is  preparing  a Systen 
Development  Life  Cycle  Manual,  from  which  the  headings  were  taken. 


The  mathenacical  model  of  project  manpower  consists  of  the  equa- 
tion for  each  cycle,  plus  a linking  function  which  specifies  the 
relative  sizes  and  durations  of  the  cycles  and  their  lags  or 
spacing  in  calendar  time.  Typical  inter-cycle  ratios  are  given  i.n 
Exhibit  I,  p.26,  of  Appendix  B.  The  linking  relationships  have 
been  encouragingly  stable  over  a wide  range  of  projects.  This 
fact  makes  it  possible  to  develop  projections  of  manpower  and  time 
requirements  for  comparable  projects,  give.n  a few  actual  points  on 
the  earliest  cycle.  In  addition,  early  warning  of  significant 
departures  from  current  schedules  can  be  obtained  by  monitoring 
one  deviation  of  actual  manpower  utilization  from  the  established 
plan. 

Some  Repent  Experiences 

There  are  three  observations,  arising  out  of  recent  experiences 
with  life  cycle  applications,  which  are  worthy  of  note: 

Successful  projects  - these  were  generally  associated  with  rela- 
tively high  ' crashiness ' . i.e.  their  cycles  were  of  a pronounced 
'sugar loaf  shape,  and  peaked  early.  Such  a shape  may  be  expected 
where  you  have  an  experienced  project  manager  and/or  project  team, 
a high  sense  of  urgency,  eind  requisite  resources  readily  at  hand. 
Piggy-back  cycles  - these  are  secondary  'bumps'  on  t.he  projected 
primary  cycles.  They  are  generally  associated  with  changes  of 
project  scope  ( or  re-engineering  due  to  technical  'glitches'), 
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and  when  plotted  in  isolation  (i.e.  as  residues  from  the  prinary 
cycle  projection,  on  which  they  are  superimposed)  tend  tc  have  the 
shape  of  a standard  Weibull  or  Rayleigh  cycle.  Such  secondary 
cycles  should  not  have  their  .manpower  included  v;ith  the  pri.mary 
cycle  cn  which  they  ride,  when  usi.ng  one  inter-cycle  ratio  projec- 
tions, if  they  are  due  only  ro  rework . Change-of-scope  incre- 
ments, however,  legitimately  add  to  the  total  amount  of  work  to  be 
done,  and  should  be  added  in. 

Software  Development  'Single  Cycle'  Phenomenon  - This  is,  perhaps, 
the  most  significant  recent  observation:  Unlike  hardware  projects, 
in  which  the  sum  of  the  individual  cycle  curves  rarely  adds  up  to 
a 'pure'  Weibull  shape,  software  project  development  yields  cycles 
that  generally  do.  This  would  imply  that  software  developm.ent  is 
carried  out,  despite  the  fact  that  crude  stages  have  been  identi- 
fied in  it,  as  a tightly  interlaced,  functionally  homogeneous 
effort.  In  Life-Cycle  parlance,  we  could  call  it  single-purpose. 
Perhaps  this  is  due  to  the  fact  that,  particularly  with  on-line 
coding  and  debugging,  coding,  testing,  re-coding,  re-testing,  etc. 
are  done  at  such  a 'micro'  level  throughout  the  project,  that  the 
boundaries  of  macro-phases  are  lost  and  the  whole  job,  de  facto, 
is  a homogeneous  problem-solving  effort. 

One  last  observation  may  be  worth  adding.  Table  2 shows  the 
influence  of  modern  programming  productivity  aids  on  the  size  and 
shape  of  project  effort:  The  use,  in  this  case,  of  Decision  Table 
Processors  has  resulted  in  the  reduction  of  total  man-hours , in  a 
controlled  experiment  involving  a 492-mh  project,  by  some  60%,  and 
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APPUCATION  OF  THE  UFE-CYCLE  MODEL  ON  A COMPLEX  DEVELOPMENT 

PROTECT:  A CASE  HISTORY 


Introduction: 

The  Life-Cycle  Model  Is  an  econometric  model  which 
describes  the  rate  at  which  human  effort  is  applied  to  the  solution 
of  complex  problems.  Since  many  development  projects  can 
be  construed  as  an  organized  effort  aimed  at  the  solution  of  a group 
of  related  problems,  the  model  can  be  shown  to  aid  in  forecasting 
the  time  and  effort  required  to  complete  such  projects. 

The  principal  aim  of  this  paper  is  to  discuss  a specific 
application  of  this  model  on  a project  in  the  Development  Laboratory 
of  IBM's  former  Data  Systems  Division.  First,  however,  a brief  description 
of  the  Life-Cyc^e  Model  Itself. 

Life-Cycle  Model:  Theory  of  the  Model 

The  Life-Cycle  equation  is: 

-ar2 

y‘  =*  2Kate 

where: 

y*  » the  manpower  required  in  time  period  ^ This  is  usually 
stated  in  quantitltes  related  to  the  time  period  used  as 
a base;  for  example,  man-months  per  month. 

K * the  total  manpower  required  by  the  cycle,  stated  in 
the  same  units  as  ^ 
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a = a parameter,  defir.^jd  by  the  point  in  time  at 
which  y'  reaches  its  maximum  value, 
t = time,  in  equal  units  such  as  weeks  or  months, 
.measured  from  the  start  of  the  cycle, 
e = an  irrational  number  2,  7128. . . , the  base  of  the 
natural  logarithms. 

This  equation  produces  curves  such  as  those  shown  in 
Figure  1.  The  upper  portion  of  this  Figure  demonstrates  the  shape 
of  the  curve  when  j^is  held  constant,  and  a is  allowed  to  vary. 

Using  the  curve  with_a  equal  to  0.  0200  as  a standard,  an  increase 
in  the  value  of_^to  0.  0555  shifts  the  peak  of  the  curve  from  the 
fifth  month  to  the  third  month,  and  pulls  the  effective  end  of  the 
curve  hrom  the  eighteenth  month  back  to  the  eleventh  month. 
Conversely,  a decrease  in  the  value  of  a from  0.  0200  to  0.  0102 
moves  the  peak  to  the  seventh  month  and  the  effective  end  to  the 
twenty-fourth  month.  Since  these  curves  tail  out  to  infinity,  the 
effective  end  is  considered  to  be  that  time  period  where  manpower 
drops  below  one  man-month. 

The  lower  half  of  Flgtire  1 shows  the  effect  of  a change  in 
the  value  of  K when  a is  held  constant.  The  shape  of  the  curve,  as 
determined  by  its  peak  and  effective  end,  remains  constant,  but  its 
height  changes  in  proportion  to  the  value  of  K.  If  K were  set  to  1, 
the  result  would  be  the  dimensionless  type  of  curve  tabled  in  various 
handbooks. 
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The  integral  of  the  Life-C-’cle  equation  is: 
y = K(i-e  ; 

where: 

y = the  cumulative  manpower  used  through  month  t. 

K = the  total  manpower  required  by  the  cycle, 
a = a parameter  determined  by  the  time  period  in 

which  the  derivative  of  ^ reaches  its  maximum, 
t = time  in  equal  units  counted  from  the  start  of  the 
cycle 

e = 2.7128,  the  base  of  the  natural  logarithms. 

This  integral  equation  can  be  rewritten  thus: 

e-at^  , (K-y)/K  (3) 

Then  the  Life-Cycle  equation  can  be  written  in  this  form: 

y'  = 2at  (K-y)  (4) 

This  differential  equation  may  be  interpreted  as  follows. 
The  amount  of  manpower  required  in  any  given  month  depends  upon 
the  pace  of  the  work  (2at)  and  the  amount  of  work  left  to  be  done 
(K-y) . 

There  is  a more  fundamental  explanation  of  this  model, 
drawing  upon  the  theory  of  random  failures  in  life  testing,  but  its 
exposition  is  too  lengthy  for  this  paper. 
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A development  project  consists  of  one  or  more  of  these 
cycles,  each  defining  the  manpower  required  to  complete  a particular 
phase  of  the  project.  Each  cycle  has  the  same  functional  form 
although  each  cycle  will  probably  have  different  values  of  j^and  a. 
The  common  cycles  found  in  projects  at  IBM  are: 

Planning 

Design 

Prototype  Assembly  and  Test 

Release  to  Manufacturing. 

Figure  2 is  an  idealized  representation  of  these  cycles,  laid 
out  on  a time  base  as  they  might  be  in  a real  project.  Some  checlc- 
points  are  associated  with  various  cycles  to  illustrate  how  forecasts 
of  milestones  can  be  made  once  their  positions  relative  to  the  cycles 
has  been  determined. 

Fitting  the  Life-Cycle  Model  to  Data 

There  are  two  methods  of  fitting  Life-Cycle  curves  to  data. 
The  first  assumes  that  the  data  is  grouped  by  cycle.  Then  a least 
squares  solution  can  be  found  for  K and  a by  this  formula: 

In  * In  (2Ka)  - at^  (5) 

which  is  a rewritten  form  of  the  Life-Cycle  equation. 
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When  the  peak  of  the  curv3  is  known,  the  parameters  K and 
a can  be  estimated  more  readily;  ^ 

I I 

a = 1/  2t2  (6) 

^ ® max  • max ) 

When  data  is  not  grouped  by  cycle,  it  can  be  separated  into  its 
component  cycles  by  the  following  procedure: 

1.  Plot  the  data  on  a graph.  It  will  be  similar  to 
the  "Total  Projected  Man-Months  Per  Month" 
line  in  Figure  3. 

2.  At  the  point  where  the  second  cycle  starts , 
there  is  usually  a marked  increase  in  manpawer 
utilization,  as  can  be  seen  readily  in  Flgtire  3. 

3.  Fit  a Life-Cycle  curve  to  the  data  up  to,  but 
net  including,  this  point. 

4.  Compute  the  values  for  the  Life-Cycle  curve. 

5.  Subtract  these  computed  values  from  the 
observed  data,  and  construct  a new  series 
composed  of  these  residuals. 

S.  Starting  with  the  point  where  the  marked 
increase  was  observed,  repeat  steps  1 
through  5. 
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7. 


Repeat  this  procediire  until  all  cycles  have 
been  identified. 


Forecasting 

Forecasts  made  by  the  Lite-Gycla  Model  fail  Into  two 
distinct  types.  The  first  is  a forecast  made  before  the  project 
starts;  the  second  is  a forecast  made  on  a going  project. 

Both  types  of  forecasts  require  a knowledge  of  the  various 
factors  that  relate  successive  cycles.  These  factors  are: 

1.  The  ratio  of  the  total  time  requir.ed  in  one 
cycle  to  the  total  time  required  in  the  next 
cycle. 

2.  The  ratio  of  the  total  manpower  required  in 
one  cycle  to  the  total  manpower  required  in 
the  next  cycle. 

3.  Where  in  one  cycle  the  next  cycle  starts. 

These  factors  must  be  determined  individually  for  each 

installation.  Those  shown  in  Figure  4 have,  however,  proved 
useful  as  an  initial  set. 

Using  factors  such  as  these,  a Life-Cycle  forecast  can 
be  made  before  the  project  starts  by  asking  a responsible  engineer 
to  estimate: 
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1. 


When  one  of  these  cycles  will  start 


2.  The  time  required  to  reach  his  peak  effort 

3.  The  manpower  required  at  peak  effort. 

I 

' Using  equations  6 and  7 aforementioned,  the  parameters  of 

1 

1 this  cycle  are  computed.  Then,  using  the  factors  shown  in  Figure  4, 

[ the  parameters  of  all  other  cycles  are  computed.  Finally,  the  manpower 

requirements  by  cycle  are  laid  out  in  time. 

The  effectiveness  of  such  projections  can  be  gauged  from  an 

j 

experience,  not  atypical.  The  project  engineer  had  estimated  a 
^ proposed  project  would  require  twenty-five  man-years.  Based  on  his 

answers  to  the  questions  above,  Life-Cycle  forecasted  fifty-three 
.man-years.  The  engineer  raised  his  estimate,  and  the  project  started. 
On  the  basis  of  four  months  of  actual  data,  Life-Cycle  forecasted  eighty 
man-years  of  effort.  This  happended  to  be  the  identical  amount  then 
forecasted  by  the  project  manager. 

Cnee  a project  has  started,  a forecast  is  made  by  fitting 
Life-Cycle  curves  to  existing  data,  determining  the  number  of  cycles 
which  the  project  is  likely  to  have,  and  forecasting  these  by  means 
of  the  relationships  shown  in  Figure  3.  The  effectiveness  of  this 

technique  can  be  gauged  by  the  case  study  which  forms  the  second 
half  of  this  paper. 


A 


CASS  STUDY  OF  APPUCi^lION  OF  THE  UFE -CYCLE  MODEL 


Descnotion  of  the  Proiect 

The  objective  of  this  project  was  to  produce  a system  under 
contract  to  a customer  to  fill  his  particular  need  and  at  the  same  time 
to  add  a more  general  purpose  version  of  this  system  to  IBM's  regular 
product  line. 

The  system  consisted  of  three  separate  boxes  which  operated 
in  unison;  i.  e.  , no  one  box  could  process  data  usefully  without 
the  other  two.  One  of  these  boxes  contained  a large  mechanical 
device  which  was  to  be  supplied  by  a subcontractor.  The  effort  on 
this  device  was  never  included  in  the  Life -Cycle  projections. 

The  contract  system  was  scheduled  for  shipment  to  the 
customer  twenty-six  months  after  the  start  of  the  project.  The 
commercial  system  was  scheduled  for  first  shipment  thirty-six  months 
after  the  start  of  the  project. 

Organizationally,  the  project  started  in  another  Division  of 

« 

IBM  but  was  transferred  to  the  Data  Systems  Division  after  five 
months.  The  work  was  to  be  geographically  relocated  to  Poughkeepsie, 
and  an  entirely  new  set  of  personnel  assigned  to  the  project. 

Shortly  after  the  Data  Systems  Division  assumed  responsibility 
for  this  project,  the  Operations  Research  Department  interested  the 
new  project  manager  in  the  Life-Cycle  Model  and  produced  the 
projection  shown  in  Figure  S. 
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Aside  from  the  data  just  civen  on  the  project  and  its 
organization,  the  Life-Cycle  analyst  also  knew  that  the  Design 
Cycle  had  supposedly  started  in  the  fourth  month  of  the  project's 
life,  that  the  replacement  of  manpower  had  started  in  the  sixth  month, 
and  that  the  Prototype  Assembly  and  Test  Cycle  was  scheduled  to 
start  in  the  thirteenth  month. 

This  projection  indicated  shipment  of  the  first  regular  product 
line  system  about  the  time  it  was  scheduled  although  the  manpower 
forecast  was  substantially  larger  than  the  official  Engineering  Estimate 
anticipated.  The  project  manager  based  his  next  budget  request  on 
this  Life-Cycle  forecast. 

This  first  Life-Cycle  forecast  made  no  attempt  to  pinpoint 
shipment  of  the  contract  system. 

The  second  Life-Cycle  forecast  on  this  project,  shown  in 
Fig-ure  6.  was  made  six  months  later.  Several  things  had  changed. 

The  system  was  no  longer  expected  to  become  part  of  the  regular 
product  line.  The  transfer  of  manpower  had  not  occurred,  and  the 
project  continued  with  its  original  manpower  under  the  direction 
of  the  Data  Systems  DiVsion.  The  project  was  encountering  severe 
technical  problems. 

This  second  Life-Cycle  forecast  reflected  these  happenings 
in  a larger  initial  Design  Cycle  and  a secondary  Design  Cycle 
starting  In  the  eleventh  month  of  the  project's  life. 
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The  third  Life-Cycle  forecast,  shown  in  Figure  7,  showed 
no  significant  change  in  the  project's  outlook.  Shipment  was  now 
forecasted  for  the  twenty-ninth  month. 

Figure  8 compares  the  manpower  forecasts  of  the  three  Life- 
Cycle  forecasts  discussed  in  this  paper.  Even  the  first  forecast 
fell  within  the  generally  accepted  management  limits  of  pluS  or 
minus  tan  per  cent  of  the  actual  cost.  The  second  and  third  forecasts 
were  ideally  close  to  the  actual  cost. 

Figure  9 compares  the  shipment  dates  forecasted  by  these 
and  two  other  intermediate  forecasts.  Again,  all  forecasts  predicted 

I 

well  although  the  first  forecast  anticipated  a system  which  was  never 
produced. 

In  both  time  and  manpower,  Life-Cycle  forecasted  more 
accurately  than  conventional  techniques  used  in  the  Laboratory. 

The  Operations  Research  Department  learned  several  things 
from  this  project: 

1.  In  forecasting  the  various  cycles,  it  is  best 

to  use  the  average  relations  hips  between  cycles, 
rather  than  to  try  to  tailor  these  to  a particular 
project. 

The  first  forecast  attempted  to  use  factors 
based  on  one  or  two  broadly  similar  projects. 
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Use  of  average  relationships  would  have 
forecast  manpower  more  accurately. 

One  must  examine  more  detailed  figures 
(i.e.  , type  of  personnel  working) 
than  the  total  manpower  utilized  per  month 
when  monitoring  a forecast. 

The  second  forecast  might  have  been  made 
two  months  earlier  had  dus  been  done. 

Even  though  the  pattern  of  manpower 
utilization  might  shift  quite  a bit  in  time , 
the  total  manpower  and  time  that  the  project 
will  require  does  not  necessarily  change  by 
a significant  amount. 

One  implication  of  this  is  that  the  Life-Cycle 
Model  will  do  a better  job  of  forecasting 
manpower  over  a period  than  m any  one 
month. 

In  conclusion,  the  application  of  the  Life-Cycle  Model 
described  in  this  paper  proved  the  value  of  the  technique  to 
Laboratory  management.  It  was  subsequently  used  on  larger  projects 
where  management  regarded  it  as  one  of  the  numerous  indicators 
which  they  consider  when  reviewing  a project. 
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Life-Cycle  Curve:  Effect  of  Changing  Parameter  Values 
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UFE-'TsTCLE  MODEL 


Sample  Factors  Showing  the  Relationships  Between  Some  Common  Cycles 

Found  in  Development  Projects 


Manpower  and  Time  Relationships 

The  tabled  figures  show  the  relationship  of  the  manpower  or  time 
required  in  one  cycle  to  the  manpower  or  time  required  in  the  next 
cycle. 


Cycles Relationships 


Manpower 

Time 

Planning:  Design 

1:4 

1:1  1/2 

Design;  Prototype 

1:1 

1:1 

Prototype:  Release 

1:1 

1:1 

Lag  Between  the  Start  of  Successive  Cvcles 

A cycle  usually  starts  about  the  mid-point  of  the  previous  cycle's 
time  base;  i.e. , when  the  Planning  Cycle  covers  a twelve-month 
period,  the  Design  Cycle  usually  starts  in  the  seventh  month. 


Figure 
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Case  Study;  Projection  7 Mo: 
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Figure  8. 


Case  Study;  Comparison  of  Three  Projections 


year  \ I 1 YEAR  2 I I J YEAR  3 1 | YEAR  A 

TIME  OF  TIME  OF  TIME  OF  ACTUAL 

FIRST  SECOND  THIRD  SHIPMENT 

PROJECTION  PROJECTION  PROJECTION 
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Figure  9. 
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APPEND":  A 

( Chapter  references  herein  refer  to  Reference  5.  ) 

Tne  Engineering  Design  decisionmaking  process  discussed 
in  Chapter  V , leads  to  the  following  theoretical  model.  This  can 
explain  the  observed  patterns  by  asserting  that  an  engineering 
development  project  can  be  considered  as  a set  of  unsolved  prob- 
lems. 

This  unsolved  problem,  space  is  exhausted  in  the  presence  of 
hu.man  effort  and  creativity  in  a deliberate,  purposeful  group 
problem.-solving  setting.  The  operation  of  exhausting  the  prcble.m 
space  involves  ma.ny  activities  which  need  not  be  ordered  in  time. 
There  is,  however,  a partitioning  of  sets  of  these  activities.  This 
divides  the  unsolved  problem  space  i.nto  a group  of  subspaces, 
v/hich  correspond  to  the  purposes  with  which  the  proble.m, -solving 
operation  is  concerned  at  various  stages  in  the  life  of  a project. 

In  practice,  the  e:<haustion  operation  is  t.^.e  design  decision.m,aking 
operation.  In  this  context,  the  several  problem  subsets,  each 
labeled  by  its  "raison  d'etre, " represent  a stage-wise  conversion 
of  problems  into  solutions.  If  we  make  the  following  assumptions 
concerning  each  such  subset,  the  effort  utilization  patterns  actu- 
ally observed  can  be  constructed: 

1.  The  number  of  problems  in  the  subset  is  finite,  albeit 
unknown. 

2.  Human  problem-solving  effort  constitutes  an  environ- 
ment for  the  unsolved  problem  set  a.nd  makes  an  impact 
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4. 


5. 


I 
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on  the  unsolved  probleru  'n  the  set. 

IntorOicition  gathering,  gestation,  identitication  of  alternatives 
open  for  choice,  and  deliberation  consume  varying  amounts  of 
time.  Any  planni.ng  or  design  decision,  made  as  a result  of  such 
deliberations,  represents  an  event  which  causes  one  unsolved 
problem  to  be  converted  into  a solved  procle.m,  thereby  removing 
it  from  the  unsolved  proble.m  space.  If  we  assu.me  the  occurrence 
of  these  eve.nts  to  be  i.ndependent  and  random,  then  the  .^oisson 
process  would  apply,  and  an  e.xponentiaI  distribution  of  inter- 
event times  IS  a reasonable  assertion.  It  is  further  possible  to 
postulate  a conditional  probability  function  for  the  event  that  a 
problem  will  be  permanently  removed  from  the  unsolved  proble.m 
space,  given  t.hat  an  identification  event  has  occurred.  Stated  in 
another  v/ay,  we  postulate  a ccnditicnal  problem  solution,  given 
a certain  distribution  of  insightful  situations. 

T-te  para.meter  of  tne  decision  event  distribution  is  a ccm.posite, 
i.mplicit  function  of  s:<ill  levels  of  proble.m  solvers,  level  of  exer- 
tion, administrative  actions,  and  other  random  manifestations  of 
the  problem-solving  situation,  and  interaction  with  t.he  e.nvironment. 
The  number  of  people  working  in  the  engineering  group  at  any  given 
time  is  approximately  proportional  to  the  nu.mber  of  problems  "ripe" 
for  solution  at  that  ti.me.  This  leads  to  an  interesting  interpreta- 
tion; The  problem  solvers  act  in  a dichotomous  manner  as  catalysts 
of  the  problem-solving  process  as  much  as  agents  who  cause 


! 
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problems  to  be  solved. 


The  obser/ed  patterns  can  be  shown  to  follov/  from  the  above 
assumptions.  The  .model  leads  to  a surge  of  effort,  or  cycle,  several  cf 
which  have  been  shown  to  comprise  the  life  of  the  project. 

Let  us  recall  t.he  v;orl<  of  D.  L.  Marples  at  Ca.mbridge  . Ke 
postulates  that  engineering  design  decisions  have  a tree-like  structure, 
in  v;hich  technical  choices  are  .made  at  each  node,  iubject  to  som.e 
technical  criterion,  one  of  a nu.mber  of  ad.missible  alternatives  is  selec- 
ted, sym.bciically  representing  a bra.nch  which  leads  to  the  next  (problem) 
.node. 


Mow  consider  a reliability  .model  due  to  Epstein  . He  discusses 
a set  of  independent  devices  under  test,  sub)ect  to  some  environment  E, 
v/hich  is  a random  process.  Suppose  this  e.nvironment  generates  shocks, 
or  "peaks"  which  are  destructive  to  the  devices  in  question.  It  is  then 
reasonable  to  assert  that  such  peaks  (thermal  shock,  e.xtreme  vibrations, 
etc.)  are  distributed  in  a Poisson  manner  with  some  (rate)  para.meter. 
LetJ^be  a random  variable  associated  with  the  time  inter/al  between 
successive  peaks.  Then: 

Pr  (T  > t)  = Pr  (No  event  occurs  in  the  interval  0,  t. ) (1) 

where 


t = 0 is  the  time  when  the  most  recent  event  occurred. 
Then,  from  the  poisson  assumption: 


?r  (T  > t)  « e 


- A 


L 


?r  (T  ^ t)  = 1 - e 


- A t 


A>  0.  : > 0 


The  p.d.  f.  associated  v.-ith  (3)  is: 


:(t)  = X e 


-At 


This  v.’ouid  descrihe  tae  failure  distribution  in  an  all-or-none 
situation:  Given  a destructive  has  occurrac,  one  device  v/ili  rail 
with  certainty.  The  r.urcber  of  ite.T.s  rer.ainiitg,  and  the  (average)  .number 
failing  per  time  period,  v.'ill  also  follow  exponential  functions. 

y.ov!  suppose  that  the  above  situations  were  governed  by  a 
conditional  probability  of  failure  rule:  Given  that  a peak  has  occurred 


in  E,  say  that  the  probability  of  failure  is  some  consta.nt  o 
(0  ^ p ^ 1).  It  can  then  be  shov/n  that: 


?r  (T  ^ t)  = 1 


-0  Xt 


f{t)  » Ape 


-pAt 


t > 0 


This  would  result  in  a ti.me-invariant  conditional  failure  probability; 
Whenever  a shock  occurs,  a device  will  fail  with  fixed  probability.  The 
ite.ms-re.maining  and  nu-mber-failing-per-interval  tim.e  series  are  again 
exponential  curves. 

Now  consider  the  conditional  probability  of  failure  to  be  ti.T.s 
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ceper.cer.t.  It  can  then  be  demc  ctrated  that  if  the  conditional  crcbabiiity 
of  failure  is  some  function  o(t): 

?r(T  > P, 

whence:  ^ 

-r  (i.  $ t)  = i - e (3) 

and:  ^ 

f (t)  = A ? (t)  e'  ^ . e ^ (c) 


This  lea^  i to  the  class  of  Weibull  distributions,  well  knov/n  in 
reliability  work.  The  physical  i.nterpretaticn  here  v/ould  be  that  the 
probability  of  devices  succumbing  to  destructive  shocks  is  changing 
wdth  ti.me. 

In  our  case,  it  v/ill  be  recalled,  most  of  the  data  we  have  observ’ed 
can  be  tired  by: 

y * : (t)  = 1 - e (10) 

which  is  a particular  case  of  (S)  v/hen 

P (t)  = « t,  (11) 


wnere 


This  could  imply  that  our  engineers  are  learning  to  solve  problems 
with  an  effectiveness  which  is  increasing  during  each  cycle.  V/hen  one 
considers  that  familiarity  with  the  problems  at  hand  can  lead  to  greater 
i.nsight  a.nd  sureness,  this  result  is  not  implausible.  Also,  we  see  that 
parameter  ^ IS  compound:  It  consists  of  t.he  insight-generation  rate  A , 
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and  tne  solution-finding  tactor  . l;  should  also  be  noted  that  ecua- 
.iOi.  (il)  is  a special  case  of  the  family  of  learning  cun/es; 


(13) 


which  was  discussed  in  Chapter  II  (Figures  II-3,  and  4).  Substitution 

n ' • ■ • 21,  253 

wi  »-j)  a,.^  (-1)  in  (3)  produces  tne  general  W'eibuil  case 


i..e  generaii  or  3-parameter,  Vv'eiculi  density  function 


IS  given  by: 


f(x) 


(x-o)  exp  r - ia:°i]dx, 

L 0 J 


X<0( 


(14) 


Here  a may  be  considered  a location  parameter,  /3  a scale 
parameter,  and  0 a shape  parameter. 

The  general  Weibull  case  is  of  interest  in  the  present  discussion 
-ecause  lu  allows  explicit  and  general  treatment  of  the  built-in  leami.nc 
ettect  of  (13).  This  can  be  seen  .most  clearly  by  reference  to  a second 
approach  o:  tpstein  (op.  cit. ).  He  suggests  a model  based  on  the  condi- 
acnal  probability  of  failure,  or  hazard  rate  g (t),  where 

g(t)  = f(t)/  [“  1 - F(t)J  (15) 


and  shows  that  then 
F(t) 


1 - exp  g(  r ) dtij 

1 - 


: 0 (15) 
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1 s 

1 « 

vehere  G(t) 

Jo  5 ("C)  dX  . 

v/hence 

= 

g(t)  exp  1 ■ yC  g ("C)  dr  1 

t ^ 0 

L J , 

(17) 

In  our  case. 

g(t) 

= 

y'/'l-y  = 2 at 

(13) 

v/hich  car.  be  interpreted  as  a "sur^dval  ratio,  “ or  the  probability  that  a 
proble.-n  will  be  solved  ("die")  tn  the  next  time  period,  given  that  it  has 
sur/ived  (remained  unsolved)  thus  far. 

Epstein  points  out  that  the  hazard  rate  model  has  much  in  co.mm.on 
vath  the  failure  models.  In  fact,  if  the  dependence  of  g(t)  on  the  environ- 
m.ent  could  be  .made  explicit,  one  is  led  back  to  the  earlier  .model, 
equation  (3).  The  interpretation,  here,  that  g(t),  as  p(t)  in  (13), is  a 
ti.me-dependent  learning  effect,  is  an  attempt  tov/ard  .maki.ng  this 
relation  explicit.  Follov.-ing  Epstein  again,  if 

- xt  , where  k >0, 
equation  (15)  becomes 

F (t)  = 0 t<0 

= 1 - exp  [ - 

(19) 


a.nd  (17)  becomes 
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exp  f-  (-i-Y  . 

a < I \ d } 2 


This  is  the  V/eibuIl  distribution,  and  g(t)  is 
decreasing  if  G < h < I 


constant  if  k = 1 


increasing  if  );  > 1. 


In  the  Life-Cycle  cur.'e,  Sostein's 


;<  = 2 


els  ewhere 


Inis  is  consistent  tvith  our  obser/atior.s  that  engineering  group 
solution-choosing  effectiveness  (analogous  to  the  hazard  rate)  is  increas 
ing.  i.;  tne  parameter  _b  in  (13)  were  other  than  (which  leads  to  the 
aoove  k = 2),  more  general  solution-choosing  rates  would  be  ad.nitted, 
and  the  matdmum  manpower  point  would  be: 

: -^1  b - 1 

"o  \j  ab  (21) 

while  tne  points  of  inflection  of  the  manpower  cycle  would  become 


3 (b  - 1)  tV(b  - 1)  (5b  - 11 
2 ab 


These  are  the  generalized  equivalents  of  the  values  for  (10),  given 
in  Chapter  r/,  Section  F.  Equation  (22)  behaves  as  follows: 
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roots  are  imaginar/. 
all  roots  are  0;  trivial  solution, 
we  get  one  positive  root  or  one  point  of 
inilecticn. 

we  get  2 roots,  one  of  v/hich  is  positive,  the 
other  zero.  (This  is  the  Life-Cycle  o-.jn/e. ) 
we  get  2 points  of  inflection. 

V/e  decided  to  limit  practical  applications  of  the  Life-Cycle 
method  to  the  function  in  equation  (10)  because: 

1.  The  general  case  is  cumbersome  computationally. 

2.  The  general  case  requires  esti.mation  of  (at  least  one) 
additional  parameters,  which  ca.nnot  be  justified  by  the 
a.T.ou.nt  and  accuracy  of  data  points  commonly  available  in 
practice. 

a.nd 

3.  The  specific  function  chosen  appears  to  produce  projections 
adequate  for  .managerial  requirements. 


b < 1. 

b = 1, 

1 < b < 2. 

b = 2, 

b > 2. 
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APPENDIX  B 


HOW  TO  MAKE 


A LIFE-CYCLE  ANALYSIS 


Originally  prepared  by 
F.J.  O'Reilly 
formerly  with  IBM, 
Poughkeepsie  Dev't.  Labs. 


DEFINITIONS ; Cycle 

.A  cycle  is  a stage  in  the  life  of  a development  engineering  project, 
during  which  a major  portion  of  the  total  required  development  work 
is  completed.  The  project  manager  determines  the  number  of  cycles 
which  will  occur  in  his  project,  as  well  as  the  work  content  of  each 
cycle,  by  the  way  in  which  he  views  and  organizes  his  work.  However, 
most  projects  are  organized  along  the  following  lines  (see  Fig.  1)  : 

1.  Planning  Cycle 

a.  Define  the  technical  objectives  of  the  project. 

b.  Outline  the  key  technical  problems,  and  investigate 
these  in  sufficient  detail  to  show  that  a practical 
solution  is  possible. 
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c.  Establish  specifications  (i.e.  , design  objectives)  in  sufficient 
detail  to  guide  the  detailed  design  which  follows. 

d.  Perform  experimental  hardware  work  necessary  to  achieve  the 
aims  of  this  cycle. 

Z.  Design  Cycle 

All  design,  drafting,  layout,  experimental  hardware,  etc.,  neces- 
sary to  produce  the  documents  from  which  the  prototype  will  be  built. 

3.  Prototype  Cycle 

a.  Assembly  and  debugging  of  the  prototype. 

b.  Redesign  and  modification  of  existing  design  necessary  to  make 
the  prototype  perform  in  a manner  which  satisfies  the  project's 
objectives. 

c.  Engineering  test  and  evaluation. 

d.  Preparation  and  release  of  engineering  documents  to  manufacturing. 

4.  Release  Cycle 


All  engineering  laboratory  activity' on  released  documents  necessary 
to 


a.  Make  the  product  more  easily  manufacturable. 

b.  Correct  deficiencies  in  original  design. 

c.  Update  product's  technology. 

There  are  later  cycles  beyond  the  Release  Cycle,  but  their  content  has  not 
been  defined  for  the  purpose  of  Life -Cycle  analysis  at  this  writing. 

The  definitions  are,  of  course,  idealized.  They  are  offered  as  a guide  to 
the  analyst  rather  than  as  hard  and  fast  rules.  Each  situation  must  be 
weighed  and  interpreted  on  its  own  merits.  Decisions  about  the  content  of 
the  project  and  its  cycles  will  adways  remain  the  first  and  most  important 
job  of  the  Life -Cycle  analyst. 


i. 


III.  MANPOWER  UTILIZATION  CURVE 


A.  Introduction  to  the  Curve 

Tables  of  the  manpower  utilization  curve  have  been  computed.  These  appear 
in  Appendix  I. 

The  Life -Cycle  Model  describes  each  cycle  within  a project  with  the  equation 


y’  = 2 K a t e 


where 


y'  = man-months  utilized  in  any  given  month 

K = total  man-months  required  to  accomplish  the  work  in  the  cycle 
a = a coefficient  which  determines  the  month  of  peak  manpower 
t = time,  counted  in  months  from  the  start  of  the  cycle 


a constant  2.7183. 


Figure  2 illustrates  the  mechanics  of  this  equation.  Line  A,  which  represents 
the  term  2Kat,  increases  by  a constant  amount  each  month.  Curve  B represents 
. Its  value  never  becomes  zero  although  it  does  become  infinitesimally 
small.  The  combination  of  these  two  lines  produces  Curve  C which  is  the  man- 
power utilization  curve. 

This  curve  has  many  features  which  make  it  an  ideal  mathematical  model. 

Some  of  these  are  valuable  only  to  the  theoretician.  Four  of  them,  however, 
are  of  practical  value  to  the  analyst: 

1.  The  coefficient  a determines  the  month  in  which  manpower  utilization 
is  greatest.  One  can  calculate  this  month  by  the  formula 


yr 


2.  Conversely,  if  one  knows  the  month  in  which  manpower  utilization  was, 
or  will  be,  the  greatest,  he  can  calcxilate  the  coefficient  a : 


2 ( tyi 

^ max' 
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VALUE  OF  BRACKETED  EXPRESSION 


r 


I 


f 

I 


I 

f 

k 


i 


3.  If  one  knows  the  month  in  which  manpower  utilization  has  reached,  or  < 

will  reach,  a peak,  and  if  he  knows  or  can  estimate  the  manpower  level  i. 

in  that  month,  he  can  calculate  the  value  of  K,  the  total  manpower 

required  in  the  cycle; 

K = e 1/2  (y*  ) (t  , ) 

max  y max 

These  features  form  the  basis  for  the  simplest  method  of  estimating  the  coeffi- 
cients of  the  manpower  utilization  curve.  They  enable  the  analyst  to  come  up 
with  quick  results  of  considerable  accuracy.  This  technique  will  be  described 
in  greater  detail  below. 

4.  The  manpower  utilization  curve  has  a point  of  inflection,  a point  at 
which  the  decrease  in  monthly  utilized  manpower  slows  down  in  the 
descending  portion  of  the  curve.  To  locate  this  point  on  the  time  axis, 
compute 

‘infl  = 

This  point  is  usef\d  in  setting  milestones  for  a project.  When  the  cycle 
has  passed  this  point,  the  work  should  be  in  the  "clean-up"  stages. 
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B.  Methods  of  Analyzing  the  Manpower  Utilization  Curve 

The  basic  problem  in  the  analysis  of  the  manpower  utilization  curve  is  to 
estimate  the  values  of  the  parameters  K and  a.  Two  methods  of  analysis  are 
offered.  The  log  least  square  method  is  objective:  the  peak  method  is  sub- 
jective. Wherever  feasible,  one  should  use  the  objective  method. 

1 . Log  Least  Square  Method  (Example  1) 

The  form  shown  in  Example  1 has  been  specifically  designed  to  aid  in 
the  use  of  this  method,  and  the  example  has  been  chosen  to  illustrate 
the  procedure  as  follows : 

a.  On  the  top  of  the  form,  list 

Project  name 
Project  number 
Cycle  name 
Date  of  the  analysis. 

b.  Determine  the  month  in  which  the  cycle  started  and  the  latest 
month  for  which  data  is  available. 
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c.  List  these  dates  in  Column  1.  Example  I's  cycle  started  in 
September.  I960.  December,  I960,  is  the  latest  month  for 
which  data  is  available. 

d.  Column  2 has  the  months  numbered  consecutively  from  the 
first  month,  pre-printed. 

e.  In  Column  3 list  the  man-months  utilized  in  the  cycle.  If 
data  is  not  available  for  a given  month,  leave  that  line  blank. 
Disregard  the  heading  of  this  column  until  Section  IV,  Analyzing 
Project  Data. 

Note:  DO  NOT  ELIMINATE  A MONTH  because  data  is  not  available  for  it. 

In  Example  1 the  data  for  November.  1960,  the  fifth  month,  is  not  available. 

If  we  moved  the  data  for  December,  I960,  into  the  fifth  month  position,  it 
would  seriously  affect  the  answer. 

f.  If  the  data  in  Column  3 fluctuates  erratically,  it  may  be  smoothed 
by  any  of  several  common  methods,  and  the  smoothed  data 
entered  in  Column  4. 

g.  Determine  an  appropriate  scale  for  the  graph  at  the  top  of  the 
form,  and  enter  this  scale  in  the  margins  as  shown  in  Example  1. 
Label  the  scales  accordingly. 

h.  Using  small  x's,  plot  the  data  in  Columns  3 or  4 against  the 
data  in  Columns  2. 

Note:  Graphing  the  data  can  be  the  first  step  in  analysis.  However,  the 
interpretation  of  the  graph  will  be  left  to  Section  IV. 

i.  Divide  the  data  in  Column  3 (or  Column  4 if  the  data  was  smoothed) 
by  the  numbers  on  the  same  line  in  Column  2.  Enter  the  results 
on  the  same  line  in  Column  5.  Indicate  in  the  heading  of  Column  5 
whether  the  data  in  Column  3 or  Column  4 was  used. 

j.  Take  the  natural  logarithms  of  the  data  in  Column  5.  and  enter 
them  in  Column  6.  Use  four  decimal  places.  Appendix  II  tells 
how  to  take  a natural  logarithm  and  provides  tables  for  this 
purpose. 

k.  Square  the  numbers  in  Column  2,  and  enter  the  result  on  the 
same  lines  in  Column  7.  It  is  necessary  to  do  this  only  for  lines 
which  have  data  entered  in  Column  6. 

l.  Add  the  data  in  Column  6,  and  enter  the  sum  on  line  A in  the 

box  under  the  lower  right-hand  corner  of  the  graph.  In  Example  1 
this  sum  is  S.6137. 


m.  Multiply  the  data  on  each  line  in  Column  6 by  the  data  on  the 
same  line  in  Column  7;  add  the  products,  and  enter 

this  sum  on  line  B.  No  column  is  provided  for  the  product  of 
Column  6 and  Column  7,  since  one  can  multiply  and  add  to  the 
result  of  a previous  multiplication  by  using  the  accumxilate- 
multiply  feature  on  most  calculators.  Before  starting  this 
operation,  it  is  advisable  to  clear  and  then  lock  the  dials  of 
the  calculator  so  that  the  total  cannot  be  removed  acdidently. 
In  Example  1 this  sum  of  the  products  is  58.  3754. 

n.  Add  the  data  in  Column  7,  and  enter  the  sum  on  Line  C.  In 
Example  1,  this  sum  is  66.  0. 

^ o.  Square  the  data  on  each  line  in  Coltimn  7,  add  the  results, 

and  enter  the  total  on  line  D.  The  calcxilating  procedure  is 
the  same  as  that  outlined  in  Step  m.  In  Example  1 , this  sum 
* is  1650.  0. 

p.  Count  the  entries  in  Column  6,  and  enter  this  count  on  line  E. 
In  Example  1 this  count  is  5.  0. 

/ q>  To  calculate  the  value  of  the  coefficient  a,  multiply  line  A by 

line  C.  Leave  the  result  in  the  dials  of  the  calculator. 


i , Multiply  line  B by  line  E,  using  the  negative -multiply  feature. 

Enter  the  result  in  the  space  provided.  In  Example  1 this 
^ figure  is  7T.  6272.  This  figure  must  be  positive. 


Note;  If  the  result  is  negative,  check  the  positioning  of  the  decimal  point. 

■ This  positioning  must  be  consistent  throughout  these  operations.  If  the  decimal 

• i point  was  positioned  correctly,  the  difficxilty  is  in  the  data.  Stop  the  analysis 
j ^ at  this  point,  and  refer  to  Section  IV  for  remedial  procedures. 

s'**  »•  Multiply  line  D by  line  E.  Leave  the  result  in  the  dials. 

I Square  line  C,  using  the  negative -multiply  feature.  Eater  the 

* result  in  the  space  provided.  In  Example  1 this  figure  is  3894.  0. 


u.  Divide  the  result  of  Step  r by  the  result  of  Step  s.  Carry  the 
answer  to  four  decimal  places.  This  is  the  value  of  the  coeffi- 
cient a.  In  Example  1 it  is  0.  0199. 

V.  Double  the  value  of  the  parameter  a,  and  write  this  figure  under 
the  square  root  sign  and  as  the  denominator  of  the  equation  for 
K.  In  Example  1 this  figure  is  0.0398. 
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i. 

w.  Divide  1.  0 by  the  value  under  the  square  root  sign,  and  take 

the  square  root  of  the  result  to  two  decimal  places.  In  Example  \ ^ 
1 this  result  is  5.  00.  This  is  the  month  in  which  utilized  man* 
power  will  be  the  greatest.  Round  this  figure  to  the  nearest 
quarter,  e.  g.  , 5.  07  becomes  5.  00;  5.  47  becomes  5.  50;  5.  63 
5.75;  5.  88  becomes  6.  00.  Look  up  this  rounded  figure  in 
Appendix  J on  the  line  labelled  "Month  of  Maximvim  Manpower.  " i 
The  value  of  a,  to  be  used  in  computing  the  curve,  is  located  t,  i 

on  the  line  directly  beneath  it.  Copy  this  value  into  the  block 
labelled  "For  Computing  the  Calculated." 

X.  Multiply  the  coefficient  a,  as  calculated  originally  (in  Example 
1,  0.  0199),  by  line  C.  Add  line  A to  this  result,  and  enter  the 
result  in  the  space  provided.  In  Example  1 this  figure  is  6.9271.  * 

y>  Enter  line  E in  the  denominator  of  this  expression,  and  divide. 

The  result  of  this  division  is  a natural  logarithm.  Refer  to 
Appendix  II  to  take  the  anti -log  of  this  number.  Enter  this 
anti-log  as  the  numerator  of  the  next  fraction.  In  Example  1 
the  anti-log  is  3.99. 

z.  Divide  this  anti-log  by  the  value  of  2a,  entered  previously.  \ 

The  result  is  the  coefficient  K.  Round  this  figure  off  to  a 
whole  number,  and  enter  it  in  the  box  labelled  "For  Computing 
the  Calctilated. " In  Example  1,  K eqxials  100.3,  rounded  off 
to  1 00.  0. 


(1)  Refer  to  Appendix  I,  the  same  coliunn  as  in  Step  w, 
and  compute  Column  8.  Follow  the  directions  given 
in  the  Appendix. 


(2) 

(3) 


Plot  Column  8 against  Column  2,  using  small  circles. 
Initial  or  sign  the  form  to  show  who  made  the  analysis. 


2.  Peak  Method 


As  explained  in  part  A of  this  section,  one  can  estimate  the  coefficients 
K and  a if  he  can  establish  the  month  in  which  manpower  utilization  is 
at  a maximum,  as  well  as  the  level  of  this  manpower.  The  peak  method 
uses  these  estimates  in  a trial  and  error  method  of  curve  fitting.  The 
form  shown  in  Example  2 has  been  developed  to  aid  in  the  use  of  this 
method. 


a.  Plot  the  man-months  utilized  against  the  corresponding  month 

in  which  they  were  utilized.  Use  small  x's  as  shown  in  Example 

2. 
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b.  If  the  data  exhibits  an  obvious  peak,  note  the  month  in  which 

it  occurs.  This  is  the  value  of  t i 

y max 

c.  Determine  the  man-months  utilized  at  this  peak.  This  is 
y'  max* 

Note:  The  peak  may  seem  to  occur  between  two  plotted  points.  Then  take 
t y'niax  mid-point  between  the  two  t values  of  the  plotted  points.  Take 

y'max  ***  slightly  greater  than  the  y's  observed.  In  example  2,  t , is 
taken  as  5.  0,  y'max  ^2.  0.  ^ max 

d.  Multiply  y-„^^  by 

e.  Multiply  the  result  by  1.65.  The  result  of  this  calculation 
is  the  value  of  K.  Note  it  as  indicated  on  the  form. 

f.  Look  up  the  value  of  ty<  on  the  line  labelled  "Month  of 
Maximum  Manpower"  in  Appendix  I. 

g.  Calculate  and  plot  the  curve  on  the  form  on  which  the  actxials 
are  plotted,  using  Appendix  I.  Use  small  circles  as  in 
Example  2.  Calculate  the  curve  only  up  to  the  last  point  of 
actual  data. 

h.  Inspect  the  result.  If  the  x's  and  circles  do  not  follow  each 
other  closely  enough,  adjust  the  values  of  y'_-v  and  tvi  _ , 
and  repeat  Steps  a through  h. 

Note:  A change  in  K changes  only  the  height  of  the  curve.  A change  in  a 
shifts  the  peak  of  the  curve  along  the  time  axis. 

1.  When  a satisfactory  fit  has  been  obtained,  copy  the  value  of  the  coeffi- 
cient a from  the  appropriate  column  in  Appendix  I.  Calculate  the 
entire  curve. 

The  peak  method  is  the  easiest  method  rf  analysis  to  learn  and  apply.  Its 
primary  advantages  are  the  ease  and  speed  with  which  it  can  be  used,  and  its 
ability  to  handle  situations  where  an  objective  analysis  is  impossible.  The 
accuracy  of  this  method,  however,  depends  mainly  on  the  analyst's  experience 
with  the  Life -Cycle  model,  as  well  as  his  familiarity  with  the  project  which  he 
is  analyzing. 
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IV.  analyzing  project  data 

Section  III  described  the  mechanics  o£  fitting  the  manpower  utilization  curve 
to  the  data  of  a cycle.  The  analyst's  problem,  however,  is  the  interpretation 
of  his  data,  not  its  mere  mechanical  analysis.  This  section  will  establish 
some  ground  rules  to  guide  the  collection  and  the  interpretation  of  data. 

A.  Gathering  Data 

The  collection  of  data  may  seem  to  be  a simple  matter.  In  practice,  however, 
it  can  account  for  75  to  90  percent  of  the  time  required  to  complete  an  analysis. 

The  following  general  rules  should  aid  in  the  first  steps  of  data  collection: 

1.  Determine  the  appropriate  unit  of  time  to  use  for  the  analysis -day, 
week,  month,  quarter,  etc.  The  month  is  recommended  for  use  in 
engineering  development  projects  that  last  a year  or  more. 

Z,  Determine  the  unit  of  effort.  If  the  unit  of  time  is  the  month,  the  unit 
of  effort  should  be  the  man-month.  If  the-  unit  of  time  is  the  week,  the 
unit  of  effort  should  be  the  man-week.  Keeping  the  unit  of  effort  con- 
sistent with  the  unit  of  time  avoids  spurious  fluctuations  in  the  data, 
which  conceivably  could  seriously  affect  the  analysis.  Dollars  are  not 
recommended  as  a unit  of  effort. 

3.  Define  the  project.  Follow  the  ground  rules  set  out  in  Section  IL 
Never  exclude  a portion  of  the  effort  simply  because  it  is  funded  or 
accotinted  for  differently  than  the  rest  of  the  project. 

To  illustrate:  Two  identical  models  of  a unit  are  built  simultaneously.  One  is 
assembled  and  tested  by  Laboratory  personnel  who  use  Laboratory  funds.  The 
other  is  assembled  and  tested  by  Manufacturing  personnel  who  use  Manufacturing 
funds.  The  stated  purpose  of  the  work  on  the  second  unit  is  to  accelerate  man\iiact- 
uring  Learning.  However,  the  testing  also  provides  a significant  amount  of  feed- 
back to  the  Laboratory  group,  which  affects  development.  The  work  on  the  second 
unit  should  be  figured  into  the  data  for  analysis. 

4.  Determine  what  cycle  or  cycles  the  project  is  in.  Use  the  definitions 
in  Section  II  to  make  this  decision. 

5.  Obtain  all  project  numbers,  shop-order  numbers,  etc.,  which  describe 
the  project,  as  well  as  any  numbers  used  to  identify  a piece  of  work  to 
be  excluded  from  the  analysis. 

6.  The  accounting  department  is  the  primary  source  of  all  data  on  a 
project's  manpower  utilization. 


I L 
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7.  L-ife-Cycle  analysis  requires  data  to  be  summarized  at  two  levels: 

a.  The  man>months  which  the  project  has  utilized,  by  cycle, 

b.  The  man-months  which  the  project  has  utilized,  by  detailed 
totals  at  the  various  levels  of  the  account  number.  For 
this  purpose  the  "Department  Working"  code  may  be  consid- 
ered part  of  the  account  number. 

To  illustrate:  Consider  the  account-code  structure  of  the  Development  Lab- 
oratory in  1961 : 
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- XX 

- X - 
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- XXX 

A listing  of  all  charges  to  the  department  level  of  detail,  as  well  as  totals  for 
each  level  above  this  is  required. 


The  data  of  7 (a)  is  the  data  used  for  curve  fitting.  The  data  of  7 (b)  is  used 
when  necessary  to  explain  excessive  fluctuations. 

Note:  If  it  is  impossible  to  organize  data  by  cycle,  one  may  take  a total  of  the 
man-months  utilized  in  all  cycles  in  a given  month  and  analyze  this  by  a tech- 
nique explained  in  part  C of  this  section. 

8.  If  the  data  outlined  in  the  previous  step  is  published  in  a report  form, 
the  analyst  should  familiarize  himself  with  the  procedures  followed 
in  preparing  the  report  to  make  sure  he  understands  exactly  what  the 
figures  represent. 

Of  the  eight  steps  outlined  above.  Step  3 is  the  most  likely  to  cause  difficulty. 
If  the  analyst  defines  the  project  inaccurately,  he  must  revise  his  data  and 
start  his  analysis  from  the  beginning. 

B.  Inside -Out  Analysis 

Inside-out  analysis  is  the  preferred  method  of  Life-Cycle  analysis.  To  use 
this  method,  one  must  have  project  data  identified  by  cycle.  The  first  step 
is  to  ascertain  that  the  data  is  valid  and  complete.  The  validity  of  the  data 
can  be  tested  by  asking  these  questions. 

1.  Are  the  methods  of  recording  data  sensitive  to  changes  in  operations 
and  cycles  within  the  project? 
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Z.  Are  the  operations  which  were  carried  out  (planning,  design,  assembly, 
test,  etc.  ) consistent  with  the  purpose  of  the  cycle? 

j 

3.  Based  on  the  nature  of  their  work,  should  the  "departments  working" 

I charge  this  cycle? 

; 4.  Does  the  most  recent  data  submitted  reflect  the  current  status  of  the 

I project  accurately? 

If  any  charges  are  questionable,  one  should  talk  to  the  project  administrator 
or  project  manager  about  them.  These  charges  may  want  to  be  transferred 
u to  another  cycle.  An  example  might  be  charges  by  an  Industrial  Design  group 

I ; . to  a planning  cycle.  The  completeness  of  the  charges  may  be  tested  by  asking 

I these  questions: 

l 
I ■ 

it  ••  !•  What  other  cycle  reasonably  might  have  been  charged  with  the  work 

of  this  cycle?  Check  the  validity  of  the  data  in  both  cycles. 

Z.  Is  it  possible  that  data  rightfully  belonging  in  this  cycle  was  charged 
to  a non-cyclical  category?  (Some  activities,  such  as  those  which 
contribute  to  capitalizable  equipment,  may  be  segregated  for  adminis- 
trative reasons.  ) Test  eqxiipment  might  be  a case  in  point.  At  times 
test  equipment  construction  is  charged  to  a separate  account,  carrying 
no  activity  code.  For  purposes  of  analysis,  however,  this  may  want 
to  be  transferred  into  a cycle. 

3.  Is  the  data  on  certain  operations  complete?  Parts  fabrication  is  an 
example.  The  project  orders  a part  through  the  Purchasing  Depart- 
ment. The  work  may  either  be  done  in  a Laboratory  shop  or  sent  to 
a vendor.  In  such  a situation  it  is  best  to  eliminate  adl  charges  from 
the  Laboratory  shop,  since  their  contributions  to  the  project  are  vul- 
nerable to  arbitrary  make-or-buy  decisions. 

When  the  data  is  checked  for  validity  and  completeness,  a log  least  square 
*•  analysis  may  be  started.  When  the  data  is  plotted,  it  should  be  subjected 
*.  to  critical  examination.  Comparatively  few  points  often  suffice  to  show  the 
general  shape  of  the  manpower  utilization  curve.  (Figure  3a). 

- U another  form  emerges  (Figure  3b),  the  data  must  be  rechecked.  The  form 
(3b)  results  from  the  cycle  starting  in  the  second  half  of  a month.  Since  the 
curve  "assumes"  that  the  effort  represents  the  entire  month,  the  shape  of  the 
curve  is  distorted.  In  this  case  the  solution  was  to  eliminate  the  first  month 
and  its  data  from  the  analysis  and  to  take  the  second  month  and  its  data  as  the 
effective  start  of  the  cycle. 


I Figure  3c  shows  no  discernible  trend.  In  this  case  one  must  go  through  the 
j j log  least  square  analysis  and  evaluate  the  result.  This  evaluation  may  indicate 
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10.4 

2 

17.4 

17.4 
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7.  Again  search  for  a discontinuity  around  the  time  the  next  cycle  should 
have  started. 

8.  Analyze  the  residxials,  from  the  first  residual  to  the  residual  for  the 
month  preceding  this  last  discontinuity  (Example  3f).  Indicate  in 
Column  3 of  the  log  least  squares  sheet  that  the  data  are  residuals. 
Compute  this  cycle  also  in  its  entirety. 

9.  Subtract  the  cycle  just  calculated  from  the  previous  residuals  for 
another  set  of  residuals  (Example  3b;  Column  5 • Column  6 = Column  7). 

10.  Repeat  Steps  6 through  9 if  more  than  one  cycle  remains  to  be  analyzed. 

11.  If  only  one  cycle  remains  to  be  analyzed,  analyze  it. 


12.  Erroneous  assumptions  concerning  starting  month,  etc.  in  the  analysis 
of  an  early  cycle  can  affect  all  later  cycles.  If  the  residuals  become 
unworkable  (i.e.  , if  they  lose  all  resemblance  to  the  form  which  the 
Life -Cycle  model  leads  us  to  expect),  reconsider  the  early  part  of  the 
analysis.  An  error  in  interpretation  here  is  probably  the  cause  of  the 
difficulty. 


Making  Projections 


Once  the  analysis  of  the  project's  history  is  complete,  the  msiking  of  a projection 
is  a simple  matter.  It  consists  of: 

1.  Extending  the  existing  cycles  to  their  termination. 


2.  Computing  the  values  for  the  cycles  which  have  yet  to  start. 

Mathematically,  the  manpower  utilization  curve  approaches  zero  at  infinity. 
However,  one  can  terminate  the  curve  by  the  adoption  of  a rule  such  as  this: 


"The  cycle  ends  when  the  expected  manpower  utilization  in  a ] 

month  drops  below  one  man-month.  " | 

This  particular  rule  is  perhaps  an  ultra -conservative  one.  It  does,  however,  | 

prevent  the  introduction  of  spurious  fluctuation  into  data  being  analyzed  by  the  | 

outside -in  approach.  | 

To  compute  the  values  of  cycles  which  have  yet  to  start,  one  needs:  1 


Factors  which  express  the  ratio  of  the  total  manpower  utilized  in 
successive  cycles. 


Factors  which 
cycles. 
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3.  A rule  which  positions  the  start  o£  a cycle  on  the  time  stream. 

To  illustrate The  Design  Cycle  o£  a project  required  100  man-months,  spread 
over  1 5 months.  The  Prototype  Cycle  required  120  man-months,  spread  over 
a 12-month  period.  The  factors  which  one  obtains  £rom  this  are; 

K (Design  Cycle)  ; K (Prototype  Cycle)  = 100:  120 

or  K (Design  Cycle)  K (Prototype  Cycle)  = 1.0  : 1.2 

and  T (Design  Cycle)  : T (Prototype  Cycle)  = 15  : 12 

T (Design  Cycle)  . T (Prototype  Cycle)  = 1.00  : 0.75 

Therefore,  in  projecting  a Prototype  Cycle  from  a Design  '"ycle,  one  would 
multiply  the  tot^l  effort  in  the  Design  Cycle  (K)  by  1.  2 and  me  total  time  (T) 
by  0.  75  to  find  the  total  effort  and  total  time  in  the  Prototype  Cycle.  Pick 
the  proper  manpower  utilization  curve  from  .Appendix  I by  inspection. 

To  obtain  factors  when  first  using  Life-Cycle  analysis,  one  should  analyze 
the  historical  data  of  some  fully  completed  projects.  Exhibit  I offers  some 
average  relationships  which  one  might  expect  to  find. 


1. 


r[ 


EXHIBIT  I 


Average  Relationships  Between  Successive  Cycles 


Cycles 


Total  Manpower  (K)  Total  Time  (T) 


^ planning:  Design 

4.0 

1.4* 

Design;  Prototype 

l.O 

1.0 

prototype;  Release 

1.0 

1.0 

r Release;  Product  Support  Number  One 

0.4 

0.7 

J 


♦ This  factor  seems  to  vary  quite  widely.  Use  of  the  average  is  only  recom- 
mended if  the  Planning  Cycle  peaks  in  the  third  month  or  later;  otherwise  use 
best  available  estimates. 


D 
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At  timet,  one  finds  that  substantial  changes  in  the  item  under  development 
cause  a secondary  cycle  (illustrated  in  Figure  4).  When  computing  the  factors 
of  this  project,  the  total  manpower  for  this  stage  is  based  on  the  total  man-  ~ 

power  in  both  primary  and  secondary  cycles.  The  total  time  is  computed  from  i, 
the  start  of  the  primary  cycle  to  the  end  of  the  primary  or  secondary  cycle, 
whichever  ends  latest. 

About  the  only  difficulty  one  encounters  in  making  a projection  is  the  problem 
of  when  to  start  a cycle.  In  theory  as  well  as  in  practice,  the  project  manager 
can  exert  more  influence  on  the  time  he  starts  a cycle  than  on  its  form.  How- 
ever. it  seems  reasonable  that  he  will  try  to  start  a cycle  at  a time  when  he 
can  utilize  the  manpower  he  is  phasing  out  of  the  previous  cycle.  It  is  suggested,  • 
therefore,  that  the  analyst  start  a cycle  at  a time  when  it  can  absorb  the  man- 
power which  is  phasing  out  of  the  previous  cycle,  and  still  minimize  the  total 
manpower  utilized  in  any  given  month. 


V.  PREPARING  A LIFE -CYCLE  PROJECTION  BEFORE  A PROJECT  STARTS 

To  lay  out  a Life-Cycle  projection  before  a project  starts,  one  needs  three 
pieces  of  information: 

1.  For  any  one  cycle,  the  months  required  to  reach  peak  manpower  in 
this  cycle. 

2.  For  the  samecycle,  the  amount  of  manpower  to  be  utilized  at  p>eak  of 
cycle. 

3.  Factors  which  give  relationships  between  successive  cycles. 

While  it  is  possible  to  lay  out  this  type  of  projection  using  any  cycle  as  a base, 
the  project  manager  can  estimate  and  for  the  first  cycle,  the 

Planning  Cycle,  most  readily  because  it  is  in  the  immediate  future. 

Using  the  relationships  explained  in  Section  III-A,  the  coefficients  K and  a for 
this  base  cycle  are  readily  computed: 


K = 1.65  (v'  ) (tv<  ) 

” max  y max 


T,  the  toul  time  in  the  cycle,  is  approximately  three  times  t . .It  be 
determined  precisely  by  computing  the  curve.  ^ max 
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The  coefficients  of  ail  other  cycles  can  be  computed  by  using  the  factors 
described  in  Section  IV-D.  Positioning  the  cycles  on  the  calendar  is  done  in 
the  same  manner  as  described  in  Section  IV-D. 

It  IS  recognized  that  this  projection  is  quite  sensitive  to  the  accuracy  of  the 
project  manager's  estimate  of  y'max  for  the  base  cycle.  However 

he  should  be  able  to  concentrate  on  this  two-point  estimate  more  easily  than 
on  a total  program  estimate. 


MONTHS 


0 2 4 8 $ 10 

months 


FIGURE  4.  An  example  of  a compound  cycle,  (a)  Compound  cycle  as  it 
might  appear  in  accounting  records,  (b)  First  cycle  of  the  compound  cycle 
shown  above.  ( c)  Second  cycle  of  the  compound  cycle  shown  above;  note 
the  difference  in  the  time  scale. 


Appendix  I 


TABLES  OF  THE  MANPOWER  UTILIZATION  CURVE 


In  these  tables: 

line  1 is  the  value  of  t^i  , the  point  in  time  at  which  manpower  , 
utilization  ( y'  ) is  greatest.  These  tables  give  curves  for  2 ate 
at  quarter-month  intervals. 

line  2 gives  the  value  of  the  coefficient  a which  is  associated  with  this 

value  of  t„( 

y max 

line  3,  etc.  gives  the  percentage  values  of  the  manpower  utilization 
curve  at  one -month  intervals. 

To  compute  a specific  manpower  utilization  curve: 

1.  Find  the  proper  value  of 

2.  In  the  column  below  this  value,  starting  at  line  3,  multiply  the 
value  in  the  table  by  K,  the  total  manpower  utilized  in  the  cycles. 
The  result  is  the  monthly  manpower  utilized. 


3.  If  one  computes  y'  for  most  or  all  of  the  curve,  the  sum  of  these 
values  should  be  slightly  less  than  the  value  of  K.  This  sum  cannot, 
and  should  not,  equal  K. 
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Ti^BLES  OF  the  MPiNFO'MEF  UTILIZ-  TIOH  CUK’VB 


WOHTH  OF  MOXIMUM  MPMPOWCR 

1 . DO 

1.25 

1 .50 

1.75 

;he  vplue  of  0 is 

0 . 5000 

0 . 3200 

C . 2^2^ 

0 • 1633 

.'XXXXXXXXXXXXXXXXXXXXXXXX  T XX 

VPLUE  OF 

THE  MfihPOWEF: 

UTILIZPTION  CURVE 

1 .0 

0.6065 

0 . 46^7 

0.3559 

0.2773 

2.0 

0.2707 

0.3559 

0.3654. 

0.3399 

3.0 

0.0333 

0. 1073 

0. 1304 

0.2254 

4.0 

0.0013 

0.0153 

0.0509 

0.0953 

3.0 

0.0000 

0.001 1 

0.0086 

0.0276 

6.0 

0. 0000 

0.0009 

0.0055 

7.0 

0.0001 

0. 0008 

8.0 

0.0001 

TOBLES  OF  THE  MftNPOWER  UTILIZHTIOM  CURVE 


month  of  maximum  manpower 

2.00 

2.25 

2.50 

2.75 

THE  VALUE  OF  A IS 

0.1250 

0.0986 

0.0800 

0.0661 

XXXXXXXXXXXXXXXXXXXXXXXXX  T XX 

VALUE  OF 

THE  MANPOWER 

UTILIZATION  CURVE 

l.C 

0.2206 

0.1790 

0. 1477 

0. 1238 

2.0 

0.3033 

0.2661 

0.2324 

0.2030 

3.0 

0.2435 

0.2436 

0.2336 

0.2188 

4.0 

0. 1353 

0.1627 

0.1779 

0.1836 

5.0 

0.0549 

0.0836 

0.1083 

0.1266 

6.0 

0.0167 

0.0339 

0.0539 

0.0734 

7.0 

0.0038 

0.0109 

0.0222 

0.0363 

8.0 

0.0007 

0.0028 

0.0076 

0.0154 

9.0 

0.0001 

0.0006 

0.0022 

0.0056 

10.0 

0.0000 

0.0001 

0.0005 

0.0018 

11.0 

0.0000 

0.0001 

0.0005 

12.0 

0.0000 

0.0001 

13.0 

0.0000 

i/.'-  ...  " 

'*■  ' ' 

TpBLES 

OF  THE  mrupower 

UTiLIZmTIOm  CURVE 

WOMTH  OF  MPXIMUft  MQMPOWER 

o 

o 

• 

3.25 

3.  50 

3.75 

.HE  VQLLfE  OF  R IS 

0.0556 

0.0473 

0.0^08 

0.0356 

xxxxxxxxxxxxxxxxxxxxxxxx 

T XX 

VALUE  OF  THE  MflMPOUER 

utilizrtiom  curve 

1.0 

0.1051 

0.0903 

0.0784 

0.0686 

2.0 

0.1779 

0.  1567 

0.  1 387 

0.1234 

J.O 

0.2022 

0. 1 855 

0. 1696 

0.1549 

4.0 

0.1827 

0. 1 776 

0. 1699 

0.1610 

5.0 

0.1385 

0. 1 450 

0. 1471 

0. 1462 

• 

6.0 

0.0902 

0. 1033 

0.1127 

0.1186 

7.0 

0.0511 

0.0652  • 

0.0773 

0.0872 

8.0 

0.0254. 

0 . 0366 

0.0479 

0.0584 

• 

9.0 

0.0111 

0.0184 

0. 0269 

0.0359 

10.0 

0.0043 

0.0083 

0.01 38 

0.0203 

• 

11.0 

0.0015 

0.0034 

0.0064 

0.0106 

. ^ 

12.0 

0.0004 

0.0012 

0.0027 

0.0051 

1 J.O 

0.0001 

0.0004- 

0.0011 

0.0023 

14-.  0 

0.0000 

0.0001 

0.0004 

0.0009 

15.0 

0.0000 

0.0001 

0.0004 

16.0 

0.0000 

0.0001 

17.0 

0.0000 

TPBLES  OF  THE  MR^POWER  UTlLIZ'iTIOM  CURVE 


MONTH  OF  MRXIMUM  MfiMPOWER 
THE  VALUE  OF  A iS 


xxxxxxxxxxxxxxxxxxxxxxxxx 


4.00 

4.25 

4.50 

4.75 

0.0312 

0.0277 

0.0247 

0.0222 

T XX 

VALUE  OF 

THE  MANPOWER 

UTILIZATION  CURVE 

1.0 

o.oe06 

0.0539 

0.0482 

0.0433 

2.0 

0.1103 

0.0991 

0.0895 

0.081  1 

3.0 

0.1415 

0.1295 

0.1186 

0.1089 

4.0 

0.1516 

0.1422 

0.  1 331 

0. 1244 

5.0 

0. 1431 

0.1386 

0.1332 

0.1273 

6.0 

0. 1217 

0,1226 

0.1218 

0. 1198 

7.0 

0.0946 

0.0998 

0.1031 

0. 1047 

8.0 

0.0677 

0.0753 

0.0814 

0.0859 

9.0 

0.0448 

0.0529 

0.0601 

0.0663 

10.0 

0.0275 

0.0348 

0.0418 

0.0483 

11.0 

0.0157 

0.0214 

0.0274 

0.0334 

12.0 

0.0083 

0.0123 

0.0169 

0.0219 

13.0 

0.0041 

0.0067 

0.0099 

0.0136 

14.0 

0.0019 

0.0034 

0.0055 

0.0081 

15.0 

0.0008 

0.0016 

0.0029 

0.0045 

16.0 

0.0003 

0.0007 

0.0014 

0.0024 

17.0 

0.0001 

0.0003 

0.0007 

0.0012 

18.0 

0.0000 

0.0001 

0.0003 

0.0006 

19.0 

20.0 
21.0 

0.0000 

0.0001 

0.0001 

0.0003 

0.0001 

0.0001 

288 


r 


TABLES  OF  THE  MANPOWER  UTILIZATION  CURVE 


rOMTH  OF  MAXIMUM  MANPOWER 

?.  00 

5.25 

5.50 

5.75 

THE  VALUE  OF  A IS 

0.0200 

O.Olyt 

0.01^5 

0.0151 

XXXXXXXXXXXXXXXXXXXXXXXXX  T XX 

VALUE  OF 

THE  MANPOWER 

utilization  curve 

1.0 

0.0392 

0.0356 

0.0325 

0.0298 

2.0 

0.0738 

0.0675 

0.0619 

0.0569 

J.o 

0. 1002 

0.0924 

0.0355 

0.0792 

4.0 

0.1162 

0. 1 086 

0.1015 

0.0950 

5.0 

0.1213 

0. 1 153 

0.1093 

0. 1036 

6.0 

0.1168 

0. 1 1 3J 

0.1094 

0. 1053 

7.0 

0.1051 

0. 1 044 

0.  10  30 

0.1009 

8.0 

0.0890 

0.0909 

0.0919 

0. 0919 

o.  <2 

0.0712 

0.0751 

G.0730 

0.0800 

10.0 

0.0541 

0.0591 

0.0633 

0.0667 

1 1 .0 

0.0391 

0. 0444« 

0.0492 

0.0534 

12.0 

0.0269 

0,031 9 

0.0367 

0.  041 1 

1J.0 

0.0177 

0.0220 

0.0263 

0.0305 

14.0 

0.01 1 1 

0.0145 

0.0191 

0.0219 

15.0- 

0.0067 

0.0092 

0.0120 

0.0151 

16.0 

0.0038 

0.0056 

0.0077 

0.0101 

17.0 

0.0021 

0.0033 

0.0047 

0.0065 

iB.e 

0.001 1 

0.0013 

0.0029 

0.004-1 

T9.7 

0.0006 

O.OOlO 

0.0016 

0.0024- 

20.0 

O.OOOJ 

0.0005 

0.0009 

0.0014 

21.0 

0.0001 

0.  0003 

0.0005 

0.0008 

22.0 

2J.0 

24.0 

25.0 

0.0001 

0.0001 

0.0001 

0.0002 

0.0001 

0.0001 

0.0004 

0.0002 

0.0001 

0.0001 

TPBLES  OF  THE  MPNPOWER  UTIL IZ-":TION  CURVE 


HOHTH  OF  HPXIMUH  MRWPOWER  6.00  6.25  6.50  6.75  „ 

THE  VALUE  OF  A IS  0.01J9  0.0128  0.0118  O.OMO  L 

XXXXXXXXXXXXXXXXXXXXXXXXX  T XX  VALUE  OF  THE  MAMPOWER  UTILIZATION  CURVE 


1.0 

0.0274 

0.0253 

0.02  34 

0.0217 

2.0 

0.0526 

0.0486 

0.0451 

0.0420 

3.0 

0.0735 

0.0684 

0.0638 

0.0597 

4.0 

0.0890 

0.0834 

0.0783 

0.0737 

5.0 

0.0981 

0.0929 

0.0880 

0.0834 

6.0 

0.101 1 

0.0969 

0.0927 

0.0887 

7.0 

0.096-5 

0.0957 

0.0928 

0.0897 

8.0 

0.0914 

0.0903 

0.0688 

0.0870 

9.0 

0.0812 

0.0817 

0.0817 

0.0812 

10.0 

0.0693 

0.0712 

0.0725 

0.0732 

11.0 

0.0569 

0.0598 

0.0622 

0.0640 

12.0 

0.0451 

0.0486 

0.0517 

0.0542 

13.  0 

0.0345 

0.0383 

0.0416 

0.0447 

14.0 

0.0256 

0.0292 

0.0326 

0.0358 

15.0 

0.0183 

0.0216 

0.0248 

0.0279 

16.0 

0.0127 

0.0155 

0.0183 

0.0212 

17.0 

0.0065 

0.0108 

0.01 32 

0.0156 

18.0 

0.0056 

0.0073 

0.0092 

0.0113 

19.0 

0.0035 

0.0048 

0.0063 

0.0079 

20.0 

0.0021 

0.0031 

0.0042 

0.0054 

21.0 

0.0013 

0.0019 

0.0027 

0.0036 

22.0 

0.0007 

0.001  1 

0.0017 

0.0024 

23.0 

0.0004 

0.0007 

0.0010 

0.0015 

24.0 

0.0002 

0.0004 

0.0006 

0.0009 

25.0 

0.0001 

0.0002 

0.0004 

0.0006 

26.0 

0.0001 

0.0001 

0.0002 

0.0003 

27.0 

28.0 
29.0 

0.0001 

0.0001 

0.0001 

0.0002 

0.0001 

0.0001 

TOBLES  OF  THE  MPMPOWER  UTILIZ-TION  CURVE 


30.  C 

31. (3 

32. (3 

33.  C 


7.00 

7.25 

7.50 

7.75 

0.0102 

0.0095 

0.0099 

0.0083 

VPLUE  OF 

THE  MRNPOWER 

UTILI2PTI0N  CURVE 

0.0202 

0.0188 

0.01 76 

0.0165 

0.0392 

0 . 0366 

0.0343 

0.0322 

0.0559 

0.0524 

0.0492 

0.0463 

0.0693 

0.0654 

0.0617 

0.0583 

0.0791 

0. 0750 

0.0712 

0.0676 

0.0S48 

0.0810 

0.0775 

0.0740 

0. 0866 

0.0836 

0.0805 

0.0775 

0.0850 

0.0828 

0.0905 

‘ 0.0782 

0.0804 

0.0792 

0.0779 

0.0763 

0. 0736 

0.0735 

0.0731 

0.0724 

0.0653 

0.0662 

0.0667 

0.0669 

0.0563 

0.0580 

0.059  3 

0.0603 

0.0473 

0.0496 

0.0515 

0.0530 

0.0387 

0.041 s 

0.0436 

0.0456 

0.0-308 

0.0336 

0.0361 

0.0384 

0.0240 

0,0267 

0.0292 

0.0316 

0.0182 

0.0207 

0.0232 

0.0255 

0.0135 

0.0157 

0.0190 

0.0202 

0.0097 

0.01 17 

0.01  36 

0.0157 

0.0069 

0.0085 

0.0102 

0.0119 

0.0048 

0.0060 

0.0074 

0.0089 

0.0032 

0.0042 

0.0053 

0.0065 

0.0021 

0.0029 

0.00 37 

0.0C47 

0.0014 

0.0019 

0.0025 

0.0033 

0.0009 

0.0012 

0.0017 

0.0023 

0.0005 

0.0008 

0.001 1 

0.001 6 

0.0003 

0.0005 

0.0007 

O.OOlQ 

0.0002 

0.0003 

0.0005 

0.0007 

0.0001 

0.0002 

0.0003 

0.0004 

0.0001 

0.0001 

0.0002 

0.0003 

0.0001 

0.0001 

0.0002 

0.0001 

0.0001 

0.0001 

i( 
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TRBLES  OF  THE  MflMPOWER  UTILIZ-'rTlOH  CURVE 


MONTH  OF  MPXIMUM  NPnPOWER 

8.00 

8.25 

8.50 

8.75 

THE  VPLUE  of  P is 

0.0078 

0.0073 

0.0089 

0. 0065 

XXXXXXXXXXXXXXXXXXXXXXXXX  T XX  vqlue  of  the  MRNPOWER  UTIlIZPTION  curve 


1.C 

0.0155 

0.0146 

O.Ol 37 

0.0130 

2.0 

0.0303 

0.0285 

0.0269 

0.0254 

3.0 

0.0437 

0.0413 

0.0390 

0.0369 

4.0 

0.0552 

0.0523 

0.0496 

0.0471 

5.0 

0.0643 

0.061  1 

0.0582 

0.0555 

6.0 

0.0708 

0.0677 

0.0647 

0.0619 

7.0 

0.0746 

0.0718 

0.0690 

0.0664 

8.0 

0.0758 

0.0735 

0.0711 

0.0668 

9.  0 

0.0747 

0.0729 

0.071  1 

0. 0693 

10.0 

0.0715 

0.0705 

0.0697 

0.0680 

11.0 

0.0668 

0,0664 

0.0659 

0. 0652 

12.0 

0.0609 

0.0612 

0.061 3 

0.0612 

13.0 

0.0542 

0.0552 

0.0559 

0.0563 

14.  0 

0.0473 

0.04S7 

0.0499 

0.0508 

15.0 

0.0404 

0.0422 

0.04  78 

0.0451 

16.0 

0 . 0^  ^8 

0.0358 

0.0377 

0.0393 

17.0 

0.0278 

0.0299 

0.031  8 

0.0356 

18.0 

0.0224 

0.0245 

0.0265 

0.0283 

19.0 

0.0177 

0.0197 

0.0216 

0.0235 

20.0 

0.0137 

0.0156 

0.0174 

0.0192 

21.0 

0.0105 

0.0121 

0.01 37 

0.0154 

22.0 

0.0078 

0.0092 

0.0107 

0.0122 

23.0 

0.0058 

0.0069 

0.0082 

0.0095 

24.0 

0.0042 

0.0051 

0.0062 

0.0073 

25.0 

0.0030 

0.0037 

0.0046 

0.0055 

26.0 

0.0021 

0.0027 

0.0033 

0.0041 

27.0 

0.0014 

0.0019 

0.0024 

0.0030 

28.0 

0.0010 

0.0013 

0.0017 

0.0022 

29.0 

0.0006 

0.0009 

0.0012 

0.0016 

30.0 

0.0004 

0.0006 

0.0008 

0.001  1 

31.0 

0.0003 

0.0004 

0.0006 

0.0008 

32.0 

0.0002 

0.0003 

0.0004 

0.0005 

33.0 

0.0001 

0.0002 

0,0002 

0.0004 

34.0 

o.ooot 

0.0001 

0.0002 

0.0002 

35.0 

36.0 

37.0 

0.0001 

0.0001 

0.0001 

0.0002 

0.0001 

0.0001 
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TPBLES  OF  TH£  MPNPOWER  LLTILIZ'-TION  CURVE 


HOHTH  OF  MPXIMUK  waWPOWER 

) 

. -IE  VALUE  OF  A IS 


i <xxxxxxxxxxxxxxxxxxxxxxx 


9.00 

9.25 

9.50 

9.75 

0.0062 

0.0056 

0.0055 

0.0053 

T XX 

VALUE  OF 

THE  MANPOWER 

utilization  curve 

1.0 

0.01 23 

0.0116 

0.01 10 

0.0105 

2.0 

0.0241 

0.0228 

0.021 7 

0,0206 

J.O 

0.0350 

0.0333 

0.0316 

0.0301 

4.C 

0.0447 

0.0426 

0.0406 

0.0387 

5.0 

0.0529 

0.0505 

0.0482 

0.0461 

6.0 

0.0593 

0.0568 

0.0545 

0.0522 

7.0 

0.0^9 

0.061  4 

0.0591 

0.0569 

8.0 

0.0665 

0.0643 

0.0622 

0.0601 

9.0 

0.0674 

0.0655 

0.06 37 

0.0618 

10.0 

0.0666 

0.0652 

0.0637 

0.0622 

1 1.0 

0.0643 

0.0634 

0.0623 

0.0612 

12.0 

0.0609 

0.0605 

0.0599 

0.0592 

13.0 

0.0565 

0.0566 

0.0565 

0.0562 

14.0 

6,0515 

0.0520 

0.0524 

0.0525 

15.0 

0.0462 

0.0471 

0.0478 

0.0483 

16.0 

0.0407 

0.0419 

0.0429 

0.  0438 

17.0 

0.0353 

0.0367 

0.0380 

0.0391 

18.0 

0.0301 

O.OJIT 

0.0331 

0.0344 

19.0 

0.0253 

0.0269 

0.C295 

0.0299 

20.0 

0.0209 

0. 0226 

0.0242 

0.025T 

21.0 

0.0170 

0.0137 

0.0202 

0.0217 

22.0 

0.01 37 

0.0152 

0.0167 

0.0181 

2 J.O 

0.0108 

0.0122 

0.C1 36 

0.0150 

24.0 

0.0085 

0.0097 

0,0109 

0.0122 

25.0 

0.0065 

0.0076 

0.0087 

0.0098 

26.0 

0.0049 

0.0058 

0.006S 

0.0078 

2T.0 

0.0037 

0.0045 

0.0053 

0.0061 

28.0 

0.0027 

0.0034 

0.0040 

0.0048 

29.0 

0.0020 

0.0025 

0.00 30 

0.0037 

30.0 

0.0014 

0.0018 

0.0023 

0.0028 

Jt.O 

0.0010 

0.0013 

0.0017 

0.0021 

32.0 

0.0007 

0.0009 

0.0012 

0,0015 

33.0 

0.0005 

0.0007 

0.0009 

0.001 1 

34.0 

0.0003 

0.0005 

0.0006 

0.0008 

35.0 

0.0002 

0.0003 

0.0004 

0.0006 

36.0 

0.0001 

0.0002 

0.0003 

0.0004 

37.0 

0.0001 

0.0001 

0.0002 

0.0003 

38.0 

39.0 

40.0 

41.0 

0.0001 

0.0001 

0.0001 

0.0001 

0.0001 

0,0001 

0.0002 

0.0001 

0.0001 

0.0001 
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TPBLES  OF  THE  MPNPOWER  UTILIZ*:TION  CURVE 


POriTM  OF  HPXIMUH  NPMPOWER 
THE  VPLUE  OF  P IS 


10.00 

10.25 

10.50 

10.75 

0.0050 

0.0048 

0,0045 

0.0043 

X 

X 

VPLUE  OF 

THE  MPNPOWER 

UTILI2RTI0N  CURVI 

1.0 

0.0100 

0.0095 

0.0090 

0.0096 

2.0 

0.0196 

0.0187 

0.01 79 

0.0170 

3.0 

0.0287 

0.0274 

0.0261 

0.0250 

4.0 

0.0369 

0.0353 

0.0737 

0.0723 

5.0 

0.0441 

0.0423 

0.0405 

0.0388 

6.0 

0.0501 

0.0481 

0.0462 

0.0444 

7.0 

0.0548 

0.0528 

0.0509 

0.0490 

8.0 

0.0581 

0.0562 

0.0543 

0.0525 

9.0 

0.0600 

0.0583 

0.0565 

0.0549 

10.0 

0.0607 

0.0591 

0.0576 

0.0561 

n.o 

0.0601 

0.0589 

0.0576 

0.0564 

12.0 

0.0534 

0.0576 

0.0566 

0.0557 

13.0 

0.0558 

0.0554 

0.0549 

0.0541 

14.0 

0.0525 

0.0524 

0.C522 

0.0519 

15.  0 

0.0487 

0.0489 

0.0490 

0.0490 

16.0 

0. 0445 

0.0450 

0.0454 

0.0457 

17.0 

0. 0401 

0.0409 

0.0416 

0. 0421 

18.0 

0.0356 

0.0367 

0.0376 

0.0383 

19.0 

0.0313 

0.0324 

0.0335 

0.0345 

20.0 

0.0271 

0.0284 

0.0296 

0.0307 

21.0 

0.0232 

0.0245 

0.0259 

0.0270 

22.0 

0.0196 

0.0209 

0.0222 

0.0235 

23.0 

0.0163 

0.0177 

0.0199 

0.0202 

24.0 

0.0135 

0.0147 

0,0160 

0.0172 

25.0 

o.ono 

0.0122 

0.01 33 

0.0145 

26.0 

0.0089 

0.0099 

0,01 10 

0.0121 

27.0 

0.0071 

0.0080 

0.0090 

0.0100 

28.0 

0.0056 

0.0064 

0.0073 

0.00S2 

29.  0 

0.0043 

0.0050 

0.0059 

0.0066 

30.0 

0.0033 

0.0039 

0.0046 

0.0053 

31 . 0 

0.0025 

0.0030 

0.0036 

0.0042 

32.0 

0.0019 

0.0023 

0.0029 

0.0033 

33.  0 

0.0014 

0.001 8 

0.0021 

0.0026 

34.0 

0.0011 

0.0013 

0.0016 

0.0020 

35.0 

0.0008 

0.0010 

0.0012 

0.0015 

36.0 

0.0006 

0.0007 

0.0009 

0.001  1 

37.0 

0.0004 

0.0005 

0.0007 

0.0009 

38.0 

0.0003 

0.0004 

0.0005 

0.0006 

39.0 

0.0002 

0.0003 

0.0004 

0.0005 

40.0 

0.0001 

0.0002 

0.0003 

0.0003 

41.0 

0.0001 

0.0001 

0.0002 

0.0002 

42.0 

0.0001 

0.0001 

0.0001 

0.0002 

43.0 

44.0 

45.0 

0.0001 

0.0001 

0. 

0.0001 

0.0001 

0. 
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tables  of  IHE  MPHPOtriER  UTILIZ-^TION  CURVE 


WOWTH  OF  MAXIMUM  MANPOWER 

o 

o 

• 

11.25 

11.50 

11.75 

THE  VALUE  OF  A IS 

0.004» 

0.0040 

0.0038 

0. 0036 

XXXXXXXXXXXXXXXXXXXXXXXXX  T XX 

VALUE  ^ 

THE  MANPOWER 

UTILIZATION  CURVE 

1.0 

0.0082 

^^0.0079 

0.0075 

0.0072 

2.0 

0.0163 

0.0156 

0.0149 

0.0143 

3.0 

0.0239 

0.0229 

0.0219 

0.0210 

4.0 

0.0309 

0.0297 

0.0285 

0.0273 

5.0 

0.0373 

0.0358 

0.0344 

0.0331 

6.0 

0.0427 

0.|^1  1 

0.0396 

0.0381 

7.0 

0.0472 

O.J|f56 

0.0440 

0.  0425 

8.  0 

0.050S 

0H91 

0.04  75 

0.  0460 

9.0 

0.0532 

0|Pl6 

0.0501 

0.0486 

10.0 

0.0547 

0.0532 

0.0519 

0.0504 

11.0 

0.0551 

0.0539 

0.C526 

0.0514 

12.0 

0.0547 

0.0537 

0.0526 

0.0516 

1 3.0 

0.0534 

0.0527 

C.0519 

0.051 1 

14.  0 

0.0515 

0.0^10 

0.0505 

0.0499 

15.0 

0.0489 

0.0487 

0.0494 

0.0481 

16.  0 

0.0459 

0.0460 

0.0460 

0.  0459 

17.0 

0.0426 

0.0429 

0.04  31 

0.  0432 

18.  0 

0.0390 

0.0395 

0.0400 

0.0403 

19.0 

0. 0353 

0.0361 

0.0367 

0 • 0372 

20.0 

0.0317 

0.0325. 

0.0333 

0. 0340 

21.0 

0.0281 

0.0291" 

0.0300 

0 . 0 j>09 

22.0 

0.0246 

0.0257 

0.0267 

0.  0276 

23.0 

0.0214 

0.0225 

C.C2  35 

0. 0245 

24.0 

0.0184 

0.0195 

0.0206 

C.  021  6 

25.0 

0.0156 

0.01 67 

0.01 79 

0.0188 

26.  0 

0. 0l 32 

0.0142 

0.015? 

0.0163 

27.0 

0.0110 

0.0120 

0.01 30 

O.OMO 

28.0 

0. 009i 

0. 0100 

0.0109 

0. 01  1 9 

29.0 

0.0074 

0.0083 

0.0091 

0.0100 

30.0 

0.0060 

0. 0068 

0.0076 

0.0083 

31.0 

0.0048 

0.0055 

0.0062 

0.  0069 

32.0 

0.0038 

0.0044 

0.0050 

0.0057 

33.0 

0.0030 

0.0035 

0.0041 

0.0046 

34.  0 

0.0024 

0.0028 

0.0033 

0.0057 

35.0 

0.0018 

0.0022 

0.0026 

0.0030 

36.0 

0.0014 

0.0017 

0.0020 

0.0024 

37.0 

0.001 1 

0.0013 

0.0016 

0.0019 

38.0 

0. 0008 

U.OOIO 

0.001 2 

0.  001  5 

39.0 

0.0006 

0.0008 

0.0009 

0.  001  1 

40.0 

0.0004 

0.0006 

0.0007 

0.0009 

41.0 

0.0003 

0.0004 

0.0005^ 

0.0007 

42.0 

0.0002 

0.0003 

0.0004 

0.0005 

43.0 

0.0002 

0.0002 

0.0003  V 

0.0004 

44.  0 

0.0001 

0.0002 

0.0002 

0.0003 

45.0 

46.0 

47.0 

4a.o 

0.0001 

0.0001 

0.0001 

0.0002 

0.0001 

0.0001 

£.0002 

0.0002 

0.0001 

O.QfOl 
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TQBLES  OF  THE  MfiMPOWER  UTILIZ-;TION  CL^RVE 


homth  of  MPXIMUM  MPHPOWER 
THE  VfiLUE  OF  P IS 


xxxxxxxxxxxxxxxxxxxxxxxxx 


12.00 

12.25 

12.50 

12.75 

0. 0035 

0 , 0033 

0.0032 

0.0031 

T XX 

VPLUE  OF 

THE  MPHPOWER 

UTILI2PTI0M  CURVE 

1 .0 

0. 0069 

0. 0066 

0.0064 

0. 0061 

2.  0 

0.0137 

0.0132 

0.0126 

0.0122 

3.C 

0. 0202 

0.0194 

0.01 87 

0.0180 

4.0 

0.0263 

0.0253 

0.0243 

0.0234 

5.0 

0. 0318 

0. 0307 

0.0295 

0.0285 

6.  0 

0. 0368 

0.0355 

0.0342 

0. 0330 

7.0 

0.0410 

0.0396 

0.0383 

0.0370 

e.  0 

0. 0445 

0.0431 

0.0417 

0. 0404 

9.0 

0.0472 

0. 0458 

0.0444 

0.0432 

10.0 

0.0491 

0. 0478 

0.0465 

0. 0452 

n.c 

0. 0502 

0. 0490 

0.0478 

0. 0466 

12.0 

0.0505 

0. 0495 

0.0484 

0. 0474 

13.0 

0.0502 

0. 0493 

C.04S4 

0.0476 

14.0 

0.0492 

0 . 0 4 S 6 

0.0479 

0.0471 

15.0 

0.0477 

0.0472 

0. 0467 

0.0462 

16.  0 

0.0457 

0.0454 

0.0451 

0.0448 

17.0 

0.0433 

0.04-2 

0.0432 

0. 0430 

18.0 

0.0406 

0.0408 

0.0408 

0.0409 

19.0 

0. 0377 

0. 0380 

0.0383 

0.0385 

20.0 

0 • 0-^46 

0.0352 

0.0356 

0.0359 

21.0 

0. 0315 

0.0322 

0.0328 

0. 0353 

22.  0 

0.0285 

0.0292 

0.0299 

0.0305 

23.0 

0. 0254 

0.0263 

0.0271 

0.0278 

24.  0 

0.0226 

0.0235 

0.0243 

0.0251 

25.0 

0.0198 

0.0208 

0.0217 

0.0225 

26.0 

0.0173 

0.0132 

0.01 91 

0.0200 

27.0 

0.01 49 

0.0159 

0.0168 

0.0176 

28.  0 

0.0128 

0.0137 

0.C1 45 

0.0154 

29.0 

0.0109 

0.0117 

0.01 26 

0.0134 

30.0 

0.0092 

0.0100 

0.0108 

0.0116 

31.0 

0. 0077 

0.0084 

0. 0092 

0.0099 

32.  0 

0. 0063 

0.0070 

0.0077 

0.0084 

33.0 

0.0052 

0.0058 

0.0065 

0.0071 

34.0 

0.0043 

0.0043 

0.0054 

0. 0060 

35.0 

0.0035 

0.0039 

0.0044 

0.0050 

36.0 

0.0028 

0.0032 

0.0036 

0.004 1 

37.0 

0.0022 

0.0026 

0.00  30 

0.0034 

38.0 

0.0018 

0.0021 

0.0024 

0.0028 

39.0 

0.0014 

0.0016 

0.0019 

0.0022 

40.0 

0.001 1 

0.0013 

0.C015 

0.0018 

41.0 

0.0008 

0.0010 

0.0012 

0.0014 

42.0 

0.0006 

0.0008 

0.0010 

0.001  1 

43.0 

0.0005 

0.0006 

0.0007 

0.0009 

44.0 

0.0004 

0.0005 

0.0006 

0. 0007 

45.0 

0.0003 

0.0004 

0.0004 

0.0005 

46.0 

47.0 

48.0 

0.0003 

0.0003 

0.0003 

0.0004 

0.0003 

0.0002 
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TftBLES  Of  the  MANPOmIEP  UTIL  IZ-TIOh  CUF'./E 


r<OliTM  OF  WPXIMUM  WOMPOWER 
THE  UPLUE  OF  P IS 

i xxxxxxxxxxxxxxxxxxxxxxxx 


r 


) 00 

1 7.25 

’ 7.  50' 

1 3.  75 

0.0030 

0 , 0023 

0.0027 

0.0026. 

T XX 

VPLUE  Of 

THE  MPNPOWER 

UTILIZPTION  CURVE 

1.0 

0.0059 

0.0057 

0.0055 

0.0053 

2.  0 

0.0117 

0.0113 

0.0109 

0. 0105 

J.  0 

0.01  • 

0.0167 

0.0161 

0.0155 

4.0 

0.0226 

0.021 8 

0.0210 

0.0203 

5.0 

0.0275 

0.0265 

0.0256 

0.0248 

•S.  0 

0.031 9 

0.0308 

0.0293 

0. 0289 

7.0 

0.0358 

0. 0347 

0.0336 

0.0325 

8.  0 

0.0392 

0. 0380 

C.036S 

0.0357 

9.0 

0.0^19 

0 • 0407 

0.0795 

0. 0384 

I 0.  0 

0.0440 

0.0428 

0.041 7 

0.0406 

11.0 

0.0455 

0.0444 

0.0433 

0.0422 

12.0 

0.0464 

0. 0454 

0.0444 

0. 0434 

13.0 

0.0467 

0.O45S 

0.0449 

0.0440 

14-.  0 

0.0464 

0.0456 

0.0449 

0.0441 

15.0 

0. 0456 

0. 0450 

0.0444 

0. 04SS 

16.0 

0. 0444 

0.0440 

0.0475 

0.0430 

17.0 

Q.0428 

0.0425 

0.0422 

0.0419 

18.0 

0.0408 

0.0407 

0.0406 

0.0404 

to.o 

0 . 0386 

0.0387 

0.0337 

0.0387 

20.0 

0,0362 

0. 0365 

0.0366 

0.0367 

2T.0 

0.0337 

0.0341 

0.0344 

0 . 0346 

22.0 

0.031 1 

0.0316 

0.C320 

0.0324 

2J.0 

0.0235 

0.0290 

0. 0296 

0.0300 

24..  0 

0.0258 

0. 0265 

C. 0271 

0. 0277 

25.0 

0.0233 

0.0240 

0.C247 

0.0253 

26. 0 

0.0205 

0.0216 

C . 022  3 

0.0230 

27.0 

0.0135 

0.0193 

0.0200 

0.0208’ 

28.0 

0.  Ol  63 

0.0171 

O.Ol 79 

0.0136 

29.0 

0.0143 

0.0151 

0.01  53 

0.0166 

30.0 

0.0124 

0.0132 

O.Cl  39 

0.0147 

31.0 

0.0107 

0.0114 

0.01  22 

0.0129 

32.0 

0.0092 

0.0093 

0.0106 

0.01  1 3 

33.0 

0.007S 

0.0085 

C.0091 

0.0098 

34.0 

0.0066 

0.0072 

C.C079 

0.0035 

35.0 

0.0055 

C.0061 

0.0067 

0.0073 

36.0 

0.0046 

0.0051 

0.0056 

0.0062 

37.0 

0.0038 

0.0043 

0.0047 

0.0052 

38.0 

0.0031 

0. 0035 

C.0040 

0. 0044 

39.0 

0.0026 

0.0029 

0.C033 

0.0037 

40.  0 

0.0021 

0.0024 

C.0027 

0.0031 

41.0 

0.0017 

0.0019 

0.0022 

0.0025 

42.  0 

0.0013 

0. 0016 

0.0015 

0.0021 

43.0 

0.001 1 

0.0013 

0.0015 

0.0017 

44.0 

0.0008 

0.0010 

0.0012 

0.001 4 

45.0 

46.0 

47.0 

48.0 

0.0007 

0.0008 

0 . 0006 

0.0010 

0.0003 

0.0006 

O.OOt 1 
0.0009 
0.0007 
0.0006 

1 

1 


V 

* 
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TPBLES  OF  THE  MPiMPOWER  UTILIZRTION  CURVE 


! 

I 


! 

I 

I 


i,  . 


MONTH  OF  MAXIMUM  MANPOWER 

14.00 

14.25 

14.  50 

1 4.75 

The  VALUE  OF  A IS 

0.0026 

0.0025 

0.0024 

0.0023 

• • 

XXXXXKXXXXXXXXXXXXXXX'XXXX  T XX  VALUE  OF  THE  MANPOWER  UTILIZATION  CURVE 


1.0 

0.0051 

0.0049 

0.0047 

0.0046 

2.  0 

0.0101 

0.0098 

0.0094 

0.0091 

3.0 

0.0150 

0.0144 

0.01 40 

0.0135 

4.0 

0.0196 

0.0189 

0.0193 

0.0177 

5.0 

0.0239 

0.0232 

0.C224 

0.0217 

6.0 

0.0279 

0.0270 

0.0262 

0.0254 

7.0 

0.0315 

0.0306 

0.0296 

0.0287 

8.  0 

0.0347 

0. 0337 

0.0327 

0.0317 

9.0 

0. 0373 

0. 0363 

0.0353 

0. 0343 

10.  0 

0.0395 

0.03S5 

0.0375 

0.0365 

11.0 

0.0412 

0.0402 

0.C392 

0.0383 

12.  0 

0. 0424 

0. 0415 

0.0405 

0. 0396 

13.0 

0. 0451 

0.0422 

0.0414 

0.0405 

14.  0 

0.0433 

0 , 0426 

0.0418 

0.0410 

15.0 

0.0431 

0.0424 

0.0418 

0.0411 

16.  0 

0.0425 

0. 0420 

0.0414 

0.  0408 

17.0 

0.0415 

0.041  1 

0. 0407 

0.0402 

18.0 

0.0402 

0.0399 

0.0396 

0.0393 

19.  0 

0 . 0'^86 

0.0335 

0.0383 

0.0381 

20.0 

0.0368 

0.0368 

0.0367 

0.0367 

21.0 

0.0348 

0.0349 

0.0350 

0.0350 

22.0 

0.0327 

0.0329 

0.0331 

0.0332 

23.  0 

0.0304 

0.0308 

0.031  1 

0.0313 

24.0 

0.0282 

0.0286 

0.0290 

0.0294 

25.0 

0.0259 

0.0264 

0.0269 

0. 0273 

26.  0 

0.0236 

0.0242 

0.0248 

0.0253 

27.0 

0.0215 

0.0221 

0.C227 

0.0232 

28.  0 

0.0193 

0.0200 

0.0206 

0.0212 

29.0 

0.0173 

0.0180 

0.01  87 

0.0193 

30.0 

0.0154 

0.0161 

0.0168 

0.0174 

31.0 

0.01 36 

0.C143 

0.0150 

0.0157 

32.0 

0.0120 

0.0127 

0.01 33 

0.0140 

33.0 

0.0105 

0.01  1 1 

0.0119 

0.0124 

34.  0 

0.0091 

0.0097 

0.0103 

0.0110 

35.0 

0.007S 

0.0084 

0.0090 

0.0096 

36.0 

0.0067 

0.0073 

0.0079 

0.0084 

37.0 

0.0057 

0.0063 

0.0068 

0.0073 

38.0 

0.0049 

0.0053 

0.0058 

0.0063 

39.0 

0.0041 

0.0045 

0.0050 

0.0054 

40.  0 

0.0034 

0.0038 

0.0042 

0.0047 

41.0 

0.0029 

0.0032 

0.0036 

0.0040 

42.0 

0.0024 

0.0027 

0.0030 

0.0033 

43.0 

0.0020 

0.0022 

0.0025 

0.0028 

44.0 

0.0016 

0.0018 

0.0021 

0. 0024 

45.0 

0.0013 

0.0015 

0.0017 

0.0020 

46.0 

47.0 

48.0 

0.0012 

0.0014 

0.0012 

0.0016 
0.001 3 
0.001  1 
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TPBLES  OF  THE  MOMPOWER  UTILIZ.’-TIOM  CURVE 


HOHTH  OF  HfiXIMUM  WOMPOUER 
THE  VALUE  OF  ft  IS 


XXXXXXXXXXXXXXXXXXXXXXXXX 


j. 


15.  00 

15.25 

15.  50 

15.75 

0.0022 

0.0021 

0.0021 

0.0020 

T XX 

VALUE  OF 

THE  MANPOWER 

UTILIZATION  CURVE 

T.(T 

0.0044 

0 . 0043 

0.0042 

0 . 0040 

2.0 

0.0088 

0.0085 

0.008  3 

0.0080 

z.o 

0.01 31 

0.0127 

0.0123 

0.0119 

4.0 

0.01 72 

0,0166 

0.0161 

0.0156 

5.0 

0.0210 

0.0204 

0.01 98 

0.0192 

6.0 

0.0246 

0.0239 

0.G232 

0.0225 

7.0 

0.0279 

0.0271 

0.0263 

0.0256 

8.0 

0.0308 

0. 0300 

0.0291 

0.0233 

9.0 

0.0334 

0.0325 

0.0316 

0.0308 

to.o 

0.0356 

0.0347 

0.0338 

0.0330 

1 1 .0 

0.0374 

0. 0365 

0.0356 

0.0347 

12.0 

0.0387 

0.0379 

0. 0370 

0.0362 

1 J.O 

0.0397 

0.0339 

0.0381 

0.0373 

14.0 

0.0403 

0.0395 

0.0389 

0.0380 

15.0 

0.0404 

0.0398 

0.0391 

0.0384 

IS.O 

0.0403 

0.0397 

0.0391 

0.0385 

17.0 

0. 0398 

0. 0393 

0.0388 

0 . 0384 

18-0 

0.0389 

0.0386 

0.0382 

0.0378 

TO.O 

0.0379 

0.0376 

0.0373 

0.0370 

20.0 

0.0365 

0.0364. 

0.0362 

0.0360 

2f.O 

0. 0350 

0.0350 

0.0349 

0. 0348 

22.0 

0.0334 

0.0334 

0.0334 

0.0334 

23.0 

0.0316 

0. 031 7 

0.0318 

0.0319 

24.0 

0.0297 

0.0299 

0.0301 

0. 0303 

25.0 

0.0277 

0.0230 

0.0283 

Q.  0286 

26. 0 

0.0257 

0.0261 

0.0265 

0.0268 

27.0 

0.0237 

0.0242 

0.0246 

0.0250 

29.0 

0.021 8 

0.0223 

0.0228 

0.0232 

29.0 

0.0199 

0.0204 

0.0210 

0.0215 

30.0 

0.0180 

0.0136 

0.0192 

0.0197 

31.0 

0.0163 

0.0169 

0.0175 

0.0180 

32.0 

0.0146 

0.0152 

0.0158 

0.0164 

33.0 

0.0130 

0.0137 

0.0142 

0.0148 

34,0 

0.0116 

0.0122 

0.0128 

0.0133 

35.0 

0.0102 

0.0108 

0.01 14 

0.0119 

36.0 

0.0090 

0.0095 

O.OlOl 

0.0106 

37.0 

0.0078 

0.0034 

0.0089 

0.0094 

38.0 

0.0068 

0.0073 

0.0078 

0.0083 

39.0 

0.0059 

0.0064 

0.0068 

0.0073 

40.0 

0.0051 

0.0055 

0.0060 

0.0064 

41.0 

0.0043 

0.0047 

0.0052 

0.0056 

42.0 

0.0037 

0.0041 

0.0044 

0.0048 

43.0 

0.0031 

0.0035 

0.00  38 

0.0042 

44.0 

0.0026 

0.0029 

0.0033 

0.0036 

45.0 

0.0022 

0.0025 

0.0028 

0.0031 

46.0 

47.0 

48.0 

0.0021 

0.0023 

0.0020 

0.0026 

0.0022 

0.0019 

i 
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Appendix  II 

HOW  TO  TAKE  A NATURAL  LOGARITHM  - WITH  TABLES  - AND 
HOW  TO  CONVERT  A NATURAL  LOGARITHM  TO  A NUMBER 


To  take  the  natural  logarithm  of  a number,  using  the  tables  given  in  this 

Appendix; 

1.  If  the  number  is  between  1. 00  and  9.  99: 

a.  Look  down  the  extreme  lefthand  colvimn  of  figures  until  you 
locate  the  first  two  digits  of  the  niunber. 

b.  Follow  across  this  row  until  you  locate  the  column  heading 
corresponding  to  the  next  digit. 

c.  The  figures  at  the  intersection  of  the  row  containing  the 
first  two  digits  and  the  column  containing  the  third  digit 
are  the  natural  logarithm  of  the  number. 

To  illustrate;  Take  the  natural  log  of  5.87: 

a.  Look  down  the  extreme  left-hand  column  for  5.  8. 

b.  Go  across  this  row  to  the  column  with  7 at  the  top, 

c.  The  natural  logarithm  is  1.7699. 

2.  If  the  number  is  between  10.  0 and  99.  9: 

a.  Divide  the  number  by  10.  0. 

b.  Look  up  the  natural  logarithm  of  the  result,  which  will  be 
between  l.OO  and  9.99.  The  procedure  here  is  illustrated 
above. 

c.  Add  2.  3026  to  the  result.  2.  3026  is  the  natural  logarithm  of 
10.  0.  Adding  this  to  the  logarithm  taken  from  the  table 
multiplies  it  by  10.  0,  thus  raising  this  to  the  logarithm  of 
the  origirial  number. 

To  illustrate;  Take  the  natural  logarithm  58.  76. 

a.  Divide  58.  76  by  10,  0 to  get  5.  876. 


b.  Look  up  5.  87  as  indicated  above.  The  last  digit  (6)  may  be 
dropped  as  insignificant  to  the  context  of  Life-Cycle  analysis, 
or  obtained  by  taking  6/10  of  the  difference  between  the  log  of 
5.  87  (1.  7699)  and  the  log  of  5.  88  (1. 7716)  and  adding  this  to 
the  log  of  5.  87,  i.  e.  . 0.  0017  x 0.  6 = 0.  00102  and  1. 7699  + 

0.  0010  = 1. 7709. 

c.  Add  2.  3026  to  this  logarithm  to  obtain  the  log  of  the  original 
number,  i.  e.  , 1.  7709  + 2.  3026  = 4.  0735  which  is  the  natural 
log  of  58.  76. 

3.  If  the  number  is  between  0.  100  and  0.999: 

a.  Multiply  the  number  by  10.  0.  The  result  will  be  a number 
between  1.  00  and  9.  99. 

b.  Look  up  the  natural  logarithm  in  the  table  following  the  pro- 
cedure outlined  above. 

c.  Subtract  2.  3026  from  the  figure  given  in  the  table.  The 
result  will  be  negative.  This  is  the  natural  logarithm  of 
the  original  number. 

To  illustrate:  Take  the  natural  logarithm  of  0.  587. 


Multiply  0.  587  by  10.  0 to  get  5.  87. 

Look  up  the  natural  logarithm  (1.7699)  of  5.  87. 


c.  Subtract  2.  3026  from  this  figure: 


1.7699  - 2.  3026= -0.  6327.  This  is  the  natural  logarithm  of 
0.  587. 

4.  If  the  number  is  100.  0 or  greater,  or  0.  0999  or  less,  simply  repeat 
the  process  described  in  (2)  and  (3)  until  the  number  falls  between. 

1.  00  and  9.  99.  Add  or  subtract  2.  3026  to  the  tabled  value  the  same 
number  of  times  you  divided  or  multiplied  the  original  number  by 
10.  0 to  bring  it  within  the  limits  of  1.  00  and  9.  99. 

^ To  illustrate:  Take  the  natural  logarithm  of: 


587.  0 


1. 7699  + 2. 3026  + 2. 3026  * 6. 3751 


b.  5870.  0 = 1.7699  + 2.3026-1-2.  3026 

+ 2.3026  = 8.6777 
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c. 

0.  0587 

= 1. 7699  - 2. 3026 

- 2. 3026= 

-2.8353 

d. 

0.  00587 

= 1.7699  - 2. 3026 

- 2. 3026 

- 2. 3026 

s 

-5. 1379 

To  convert  a natural  logarithm  to  a number,  using  the  tables  given  in  this 
Appendix: 

1.  If  the  natural  logarithm  is  at  least  0.  0000  but  less  than  Z.  3026: 


a.  Find  the  logarithm  in  the  body  of  the  table. 


b.  The  first  two  digits  of  the  number  are  those  in  the  extreme 
left-hand  column  which  identify  the  row  containing  the  loga- 
rithm. The  third  digit  of  the  number  is  the  one  which  iden- 
tifies the  column  containing  the  logarithm. 

To  illustrate:  Find  the  number  whose  logarithm  is  1.7699. 


a.  Locate  the  logarithm  in  the  table. 


b.  Note  the  number  that  identifies  the  row  containg  the  logarithm 
(5.8).  These  are  the  first  two  digits  of  the  number. 

c.  Note  the  number  that  identifies  the  column  containing  the 
logarithm  (7).  This  is  the  third  digit  of  the  ntimber,  which 
is  5.  87. 


Note:  The  logarithm  you  must  convert  to  a number  will  generally  not  appear 
in  the  table.  Usually  sufficient  accuracy  can  be  obtained  by  finding  the  number 
of  the  tabled  logarithm  nearest  to  the  one  you  have.  Greater  accuracy  affects 
only  the  third  digit  of  the  number.  Greater  accuracy  can  be  obtained  by  inter- 
polating between  tabled  values. 

2.  If  the  logarithm  is  2.  3026  or  greater: 


a.  Subtract  2.  3026  from  the  logarithm  as  many  times  as  is 
necessary  to  bring  the  logarithm  within  the  range  of  the 
Uble  (0.  0000  to  Z.  3025). 

b.  Find  the  number,  following  the  procedure  given  above. 

c.  Multiply  the  result  by  10.  0 as  many  times  as  you  subtracted 
2. 3026. 


To  illustrate;  Find  the  number  whose  natural  logarithm  is  6.  3751. 
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a. 


6. 3751  - 2. 3026  = 4. 0725 


b.  4.  0725  - 2.3026  = 1.7699 

c.  1. 7699  in  the  log  of  5.  87 

d.  5.  87  X 10.  0 X 10.  0 = 587.  0,  which  is  the  niunber  whose 
natural  logarithm  is  6.  3751. 

3.  If  the  logarithm  is  less  than  0.  0000  (i.e.  . if  it  is  negative); 

a.  Add  2.  3026  as  many  times  as  is  necessary  to  make  the 
the  logarithm  a positive  number  within  the  limits  of  the 
table. 

b.  Find  the  number,  following  the  procedure  outlined  above. 

c.  Divide  this  n\imber  by  10.  0 as  many  times  as  you  added  2 
To  illustrate;  Find  the  number  whose  natural  logarithm  is  - 2.8353. 

a.  -2. 8353  + 2.  3026  = -0.  5327 

b.  -0.5327  ^ 2.3026  = 1.7699 

c.  1.7699  is  the  log  of  5.87. 

d.  5.87  10.  0 -^lO.  0 = 0.  587,  which  is  the  number  whose 

natural  log  is  -2.8353. 


1 

I 


I 

i 

• j 


. 3026. 


♦ < 

j ' 
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TPBLES  OP  MPTURPL  LOGPRITHHS 


0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

t.o 

0. 

0.0100 

0.0199 

0.0296 

0.0392 

0.0488 

0.0583 

0.0677 

0.0770 

0.0862 

«.  t 

0.0953 

0.1044 

0.1123 

0.1222 

0. 1310 

0.1 398 

0.1484 

0.1570 

0.1655 

0.1740 

t.2 

0.1823 

0. 1906 

0.1999 

0.2070 

0.2151 

0.2231 

0.231 1 

0.2390 

0.2469 

0.2546 

1.  J 

0.2624 

0.2700 

0.2776 

0.2852 

0.2927 

0.3001 

0.3075 

0. 3148 

0.3221 

0.3293 

1.4 

0.3365 

0.3436 

0.35C7 

0.3577 

0.3646 

0.3716 

0.3784 

0.3853 

0.3920 

0. 3988 

1.3 

0.4035 

0.4121 

0.4197 

0.4253 

0.4318 

0.4383 

0.4447 

0.451 1 

0.4574 

0.4637 

1.6 

0.4700 

0.4762 

0.4824 

0.4886 

0.4947 

0.5008 

0.5068 

0.5128 

0.5188 

0.5247 

1 . 7 

0.3306 

0.5365 

0.5423 

0.5431 

0.5539 

0.5596 

0.5653 

0.5710 

0.5766 

0.5822 

1.8 

0.5879 

0.3933 

0.5958 

0.6043 

0.6098 

0.6152 

0.6206 

0.6259 

0.6313 

0.6366 

1.9 

0.6419 

0.6471 

0.6523 

0.6575 

0. 6627 

0.6678 

0.6729 

0.6780 

0.6S31 

0.6831 

2.0 

0.6931 

0.6981 

0.7021 

0.7080 

0.7129 

0.7178 

0.7227 

0.7275 

0.7324 

0.7372 

2.  1 

0.7419 

0.7467 

0.7514 

0.7561 

0. 7608 

0.7655 

0.7701 

0.7747 

0.7793 

0.7839 

2.2 

0.7885 

0,7930 

0.7975 

0.8020 

0.8065 

0.8109 

0.8154 

0.8193 

0.8242 

0.8286 

2.3 

0.8329 

0.8372 

0.8416 

0.8459 

0.8502 

0.8544 

0.8597 

0.8629 

0.8671 

0.9713 

2.4 

0.8733 

0,8796 

0.8829 

0.8379 

0.8920 

0.8961 

0.9002 

0.9042 

0.9083 

0.9123 

2.3 

0.9163 

0.9203 

0.9243 

0.9282 

0. 9322 

0.9361 

0.9400 

0.9439 

0. 9478 

0.9517 

2.6 

0.9555 

0.9593 

0.9622 

0.9670 

0.9708 

0.9746 

0.9783 

0.9821 

0.9858 

0.9895 

2.7 

0. 9933 

0.9969 

1 .0006 

1.0043 

1 . 0030 

1.0116 

1 .0152 

1 .0188 

1.0225 

1.0260 

2.8 

1.0296 

1.0332 

1.0367 

1.0403 

1.0438 

1 .0473 

1.0508 

1 . 0543 

1 . 0578 

1.0613 

2.9 

1.0647 

1.0682 

1.0716 

1 . 0750 

1.0784 

1.0818 

1 .0852 

1 . 0886 

1.0919 

1.0953 

3.0 

1.0986 

1.1019 

1.1052 

1 . 1 086 

1.1119 

1.1151 

1.1184 

1 .1217 

1.1249 

1.1282 

3.  1 

1. 1314 

1.1346 

1.1379 

1 . 1410 

1 . 1 442 

1.1474 

1.1506 

1 . 1537 

1 . 1 569 

1 . 1 600 

3.2 

1.1632 

1.1663 

1. 169< 

1.1725 

1 . 1 756 

1.1737 

1.1817 

1.1848 

1 .1878 

1 . 1 909 

3.3 

1.1939 

1.1969 

1.2000 

1.2030 

1 . 2060 

1 .2090 

1 .21 19 

1.2149 

1.2179 

1.2208 

3.4 

1.2238 

1.2267 

1.2296 

1.2326 

1.2355 

1 .2384 

1.2413 

1 .2442 

1.2470 

1 .2499 

3.3 

1.2528 

1.2556 

1.2595 

1.2613 

1 . 2641 

1.2669 

1 . 2699 

1.2726 

1 . 2754 

1.2732 

3.6 

1.2309 

1.2837 

1.2865 

1.2892 

1.2920 

1 .2947 

1.2975 

1 . 3002 

1.3029 

1 . 3056 

3.7 

1 . 3083 

1.3110 

1 .31  27 

1.3164 

1.3191 

1 .3218 

1 . 3244 

1.3271 

1.3297 

1 . 3324 

3.8 

1.3350 

1.3376 

1.3402 

1.3429 

1 . 3455 

1.3481 

1 . 3507 

1 . 3533 

1 . 3558 

1 . 3584 

3.9 

1.3610 

1.3635 

1.3661 

1.3686 

1.3712 

1 . 3737 

1 . 3762 

1 . 3788 

1.3813 

1 . 3838 

4.0 

1.3863 

1.3836 

1.3912 

1.3938 

1 . 3962 

1 . 3987 

1.4012 

1.4036 

1.4061 

1.4085 

4.  1 

1.4110 

1.4134 

1.4159 

1.4183 

1.4207 

1.4231 

1 . 4255 

1 . 4279 

1.4303 

1.4327 

4.2 

1.4351 

1.4375 

1.4399 

1.4422 

1.4446 

1 .4469 

1.4493 

1.4516 

1 . 4540 

1 . 4563 

4.3 

1.4386 

1.4609 

1.4622 

1 . 4656 

1.4679 

1 .4702 

1 .4725 

1.4748 

1.4770 

1.4793 

4.  4 

1.4816 

1.4339 

1.4861 

1.4884 

1.4907 

1.4929 

1.4951 

1.4974 

1.4996 

1 .5019 

4.3 

1.5041 

1.5063 

1.5095 

1.5107 

1 . 5 1 29 

1 .5151 

1.5173 

1.5195 

1.5217 

1.5239 

4.6 

1.5261 

1.5232 

1 . 5304 

1.5326 

1.5347 

1.5369 

1 . 5390 

1.5412 

1.5433 

1.5454 

4.7 

1.5476 

1.5497 

1.5519 

1 . 5539 

1.5560 

1.5581 

1 .5602 

1 . 5623 

1.5644 

1.5665 

4.e 

1.5686 

1.5707 

1.5729 

1 . 5748 

1.5769 

1.5790 

1.5810 

1.5831 

1.5851 

1.5872 

4.9 

1.5892 

1.5913 

1.5933 

1 . 5953 

1.5974 

1.5994 

1.6014 

1 . 6034 

1 . 6054 

1 .6074 

5.0 

1.6094 

1.6114 

1.6134 

1.6154 

1.6174 

1.6194 

1.6214 

1.6233 

1 . 6253 

1.6273 

5.  1 

1.6292 

1.6312 

1.6332 

1.6351 

1.6371 

1.6390 

1 .6409 

1 . 6429 

1.6448 

1.6467 

5.2 

1.6487 

1.6506 

1.6525 

1.6544 

1 . 6563 

1 . 6582 

1 .6601 

1.6620 

1.6639 

1.6658 

5.3 

1.6677 

1.6696 

1.6715 

1 . 6734 

1.6752 

1.6771 

1 . 6790 

1 . 6808 

1.6827 

1.6845 

5.4 

1 . 6864 

1.6882 

1.6901 

1.6919 

1.6938 

1.6956 

1.6974 

1.6993 

1.7011 

1 . 7029 
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TPgLES  OF  NPTURRL  UOGPPITHMS 


0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

5.5 

1 . 7047 

1.7066 

1 . 7094 

1.7102 

1 .7120 

1.7138 

1,7156 

1 .7174 

1.7192 

1.7210 

5.6 

1 . 7228 

1 . 7245 

1.7263 

1.7281 

1 . 7299 

1.7317 

1.7334 

1 . 7352 

1 . 7370 

1.7387 

5.7 

1.7405 

1 . 7422 

1 .7440 

1 . 7457 

1.7475 

1 . 7492 

1 .7509 

1 . 7527 

1 . 7544 

1.7561 

5.8 

1 . 7579 

1 . 7596 

1.7613 

1.7630 

1 . 7647 

1 . 7664 

1.7681 

1.7699 

1.7716 

1 . 7733 

5.9 

1.7730 

1 . 7766 

1.7793 

1 . 7800 

1.7817 

1.7834 

1.7851 

1 . 7867 

1 . 7884 

1 .7901 

' • 

6.0 

1.7918 

1 . 7934 

1.7951 

1 . 7967 

1 . 7984 

1 .8001 

1 .801 7 

1 . 8034 

1 . 8050 

1 .9066 

6.1 

1 . 8082 

1.8099 

1.8116 

1.8132 

1.8148 

1 .8165 

1.8181 

1.8197 

1.8213 

1 . 9229 

6.2 

1 . 8245 

1.8262 

1.8279 

1 . 8294- 

1,8310 

1 . 8326 

1.8342 

1 . 8358 

1.8374 

1.9390 

6.  J 

1 . 8405 

1.8421 

1.8427 

1 . 8453 

1 . 8469 

1 .8485 

1 . 8500 

1.8516 

1.8532 

1 . 9547 

6.4. 

1 . 8562 

1.8579 

1 . 8594 

1.8610 

1 . 8625 

1 .8641 

1 . 8656 

1 .8672 

1 . 8687 

1 . 9703 

6.5 

1 .8718 

1.8733 

1.8749 

1 . 8764 

1.8779 

1 .8795 

1.8810 

1 . 8825 

1.8840 

1 . 9856 

6.6 

1 . 8871 

r.8886 

1.8901 

1.8916 

1 .8931 

1 .8946 

1.8961 

1 . 8976 

1.8991 

1 . 9006 

6.7 

1.9021 

1.9036 

1.9051 

1 . 9066 

1.9081 

1 .9095 

1 .91 1 0 

1.9125 

1.9140 

1.9155 

6.8 

1.9169 

1.9184 

1.9199 

1.9213 

1.9228 

1 .9242 

1 . 9257 

1 . 9272 

1 . 9286 

1 .9301 

- ■ 

6.9 

1.9315 

1 . 9330 

1.9344 

1.9359 

1 . 9373 

1.9387 

1 . 9402 

1.9416 

1 .9430 

1 . 9445 

r.o 

1 . 9459 

1 .9473 

1 . 9489 

1 . 9502 

1 .9516 

1 .9530 

1 . 9544 

1 . 9559 

1 . 9573 

1.9587 

T.1 

1.9601 

1.9615 

1.9629 

1 . 9643 

1.9657 

1 .9671 

1 . 9685 

1 . 9699 

1.9713 

1 . 972T 

7.2 

1.9741 

1.9755 

1.9769 

1 . 9782 

1 . 9796 

1 .9810 

1 .9824 

1 . 9838 

1.9851 

1 . 9865 

7.2 

1 .9879 

1.9892 

1 . 9906 

1 . 9920 

1 . 9933 

1 .9947 

1.9961 

1 . 9974 

1 . 9988 

2.0001 

« • 

7.4. 

2.0013 

2.0028 

2. 0042 

2.0055 

2.0069 

2. 0082 

2.0096 

2.0109 

2.0122 

2.0136 

7.5 

2.0149 

2.0162 

2.0176 

2.0189 

2. 0202 

2.0215 

2.0229 

2.0242 

2.0255 

2. 0268 

-- 

T.6 

2.0281 

2.0295 

2. 0309 

2. 0321 

2.  0334 

2.0347 

2.0360 

2.0373 

2.0386 

2.0399^ 

T.r 

2.0412 

2.0425 

2.0429 

2. 0451 

2.0464 

2.0477 

2.0490 

2.0503 

2.0516 

2.0528 

r.8 

2. 0541 

2. 0554 

2. 0567 

2.0580 

2.0592 

2. 0605 

2.0613 

2.0631 

2.0643 

2.0656 

T.9 

2.0669 

2.0681 

2.0694 

2.0707 

2.0719 

2.0732 

2.074.4 

2.0757 

2.0769 

2. 0782 

8.0 

2.0794. 

2. 0807 

2.0819 

2. 0832 

2. 084^ 

2.0857 

2.0869 

2.0882 

2.0894 

2.0906 

8.1 

2. 0919 

2.0931 

2. 094  3 

2.0956 

2. 0968 

2.0980 

2.0992 

2. 1005 

2.  1017 

2.1029 

8.2 

2.  1 041 

2. 1 054 

2.1066 

2. 1078 

2. 1 090 

2.1102 

2.  1 1 14 

2.1126 

2. 11 38 

2.1150 

8.2 

2.  1 162 

2.1 175 

2.1197 

2.1199 

Z.  1 211 

2. 1 223 

2.1235 

2. 1 247 

2. 1 258 

2. 1 270 

8.4 

2. 1 282 

2.  1 294 

2.1306 

2. 1318 

2. 1 330 

2. 1 342 

2. 1353 

2. 1365 

2. 1 377 

2. 1 389 

« • 

8.5 

2.1401 

2.1412 

2. 1424 

2.  1 436 

2. 1 448 

2. 1 459 

2. 1471 

2. 1483 

2. 1 494 

2.1506 

8.6 

2.1518 

2.1529 

2.1541 

2.1552 

2. 1 564 

2.1576 

2. 1597 

2.1599 

2.1610 

2.1622 

— 

8.7 

2. 1622 

2.1645 

2. 1656 

2.  1668 

2. 1 679 

2.1691 

2. 1702 

2. 1713 

2. 1 725 

2.1736 

8.8 

2. 1 748 

2.1759 

2.1770 

2.1782 

2.1793 

2.1804 

2.1815 

2.1827 

2.1838 

2.1849 

8.9 

2.1861 

2.1872 

2. 1893 

2. 1894 

2.1905 

2.1917 

2.1928 

2.1939 

2.1950 

2.1961 

9.0 

2. 1 972 

2. 1 983 

2.1994 

2.2006 

2.2017 

2.2028 

2.2039 

2.2050 

2.2061 

2.2072 

9.1 

2.2083 

2.2094 

2.2105 

2.2116 

2.2127 

2.2138 

2.2148 

2.2159 
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9.5 

2.2513 
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2.2544 

2.2558 

2.2565 
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2.2586 
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2.2607 

9.6 

2.2618 

2.2628 

2.2638 

2.2649 

2. 2659 

2.2670 
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2.2701 

2.271 1 

9.7 

2.2721 

2.2732 
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2.2752 
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2.2793 

2.2793 

2. 2803 

2.2814 

9.8 

2.2824 

2.2834 

2.2844 

2.2854 

2.2865 

2.2875 

2.2885 

2.2895 

2.2905 

2.2915 

9.9 

2. 2925 

2.2935 
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2.2956 

2.2966 

2.2976 
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2.2996 

2. 3006 

2.?016 

10.0 
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2.3036 
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2.3056 
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THE  INFLUENCE  OF  THE  TIME  - DIFFICULTY  FACTOR 
IN  LARGE  scale  SOFTWARE  DEVELOPMENT 


LAWRENCE  H.  PUTNAM 
US  Army  Computer  Systems  Connand 
Fort  Bel  voir,  VA  B2060 


ABSTRACT 

People  concerned  with  large  scale  application 
software  development  have  long  known  that  the  ability 
to  acrelerate  the  development  of  a system  even  by  add- 
ing more  manpower,  is  almost  always  unsuccessful. 

Quanti tat-ive  measures  are  developed  in  the  paper  to 
show  why  this  is  so  and  that  a system  has  a natural 
development  time  which  depends  upon  its  inherent 
difficulty. 

introduction 

Development  of  large  scale  application  software 
systems  has  been  characterized  by  severe  cost  overruns 
and  time  slippages  from  a predetermined  schedule.  This 
is  usually  because  of  the  implicit  assumption  that  the 
development  effort  is  the  product  of  people  and  time 
and  that  the  time  can  be  set  arbitrarily  by  management 
and  thus  the  manning  level  is  simply  the  development 
effort  in  manyears  divided  by  the  predetentiined  deve- 
lopment time.  Figure  1 below  illustrates  this  concept. 


It  is  clear  that  we  are  dealing  with  time  rates  of 
doing  software  work.  All  of  the  formulations  above 
assume  that  the  rates--productivity  (SsyMY)  or  manpower 
(?eople/yr)--is  constant  throughout  the  development 
period. 

It  will  be  shown  that: 

1.  These  relations  are  too  simple  to  handle  all  but 
small  programs. 

2.  Productivity  rates  are  not  constant  but  a complex 
function  of  the  system  difficulty. 

3.  Manpower  rates  are  not  constant  but  also  a function 
of  the  system  difficulty. 

It  has  been  empirically  established  by  examining 
more  than  100  large  system  (25-1000  MY  development 
effort,  2-5  years  to  develop)  that  systems  follow  a 
life  cycle  pattern  in  manpower  and  cumulative  effort 
that  look  like  this: 


MAWPOWCR 


iftontYRi 


MANPOWER 

(PEOPlt/YBl 


MANPOWER 

IPEOPIE/VR) 


CUMUIATIVE 

EFFORT 

(TOIALPEOPIE) 


T* 

i. 

r 

i 

« 
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Figure  1.  Two  different  applications  of  manpower  to  a 
100  MY  development  effort. 

Another  way  of  treating  the  problem  is  to  estimate 
the  total  number  of  source  lines  of  code  from  the 
specifications  then  to  use  some  overall  productivity 
rates  from  previous  similar  projects  and  then  divide 
the  source  code  estimate  by  the  productivity  rate  to 
get  a manyear  number.  Time  or  manpower  then  obtained 
by  a process  similar  to  figure  1.  For  example,  assume 
that  Sj  » 100,000  lines  of  source  code. 


Pir  • 1000  Sj/MY  by  analogy  with  a previous  project. 

Then  development  effort  • 100,000  , iqq 

1,000  Ss/MY 


Figure  2.  Manpower  and  cumulative  effort  as  a function 
of  time  for  software  development. 

The  data  points  are  shown  on  the  manpower  figure 
to  indicate  that  there  is  scatter  or  "noise"  involved 
in  the  process.  Empirical  evidence  suggests  that  the 
"noise"  component  may  be  up  to  ♦ 25*  of  the  expected 
manpower  value  during  the  rising"  part  of  manpower 
curve  which  corresponds  to  the  development  effort,  tj 
denotes  the  time  of  peak  effort  and  is  very  close  to 
the  development  time  for  the  system.  The  falling  part 
of  the  manpower  curve  corresponds  to  the  operations  and 
maintenance  phase  of  the  system  life  cycle.  The  princi- 
pal work  during  this  phase  is  modification,  minor  en- 
hancement and  remedial  repair  (fixing  "bugs"). 

Figure  3 shows  the  life  cycle  with  its  principal 
component  cycles  and  primary  milestones. 


i.  If  manpower  is  constrained  to  25  people  then  time 

would  be  determined  as  tj  • 100  MY  • 4 years,  or  if 

1 25  people 

! the  system  is  needed  in  two  years  then  the  relation 

a becomes  Manpower  * 100  MY  • 50  people  over  the  two 

i yn 

year  period,  or  a rate  of  50  people/yr. 

i 


I 

i 

I 

1 
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Thus,  the  life  cycle  cost  of  a system  is: 


i 


1 


CtCLI 


Figure  3.  System  Life  Cycle  model  with  component 
cycles  and  major  milestones. 

Note  that  all  the  subcycles  (except  extension) 
have  continuously  varying  rates  and  have  long  tails 
indicating  that  the  final  10'  of  each  phase  of  effort 
takes  a relatively  long  time  to  complete  (this  ex- 
plains the  "90%  done"  paradox.  A subcycle  is  90J 
done  in  work,  but  only  2/3  done  in  time). 

It  has  been  empirically  determined  that  the  over- 
all life  cycle  manpower  curve  can  be  well  represented 
by  curves  of  the  Rayleigh  form. 

y ’ • 2kate**t^  (1 ) 

where  a » 1/2  t^^  and  (2) 

K is  the  area  under  the  curve  from  t » 0 to  infinity. 

Making  the  substitution  for  a,  we  have 

y'  = K/t(j2  . t . e‘t^/2td2  (3) 

and  the  definite  Integral  of  this  is 

y = K(l-e'^^/^td^),  which  (4) 

is  the  cumulative  number  of  people  used  by  the  system 
at  any  time,  t. 

Now  we  wish  to  examine  the  parameters  K and  tj: 

K is  the  area  under  the  y curve.  It  is  the 
total  number  of  people  used  by  the  project  over  its 
1 i f e eye  1 e . 

tH  is  the  time  the  curve  reaches  a maximum.  Em- 
pirically this  is  very  close  to  the  tina  a system  be- 
comes operational  and  it  will  be  assumed  hereafter 
that  tj  » development  time  for  a system. 

Most  software  cost  is  directly  proportional  to 
people  cost  (even  computer  testing  time  is  proportion- 
al to  the  test  and  validation  manpower). 


SLC  = SCost/HY.K  (5) 

Neglecting  cost  of  computer  test  time,  inflation  over 
time,  etc.,  all  of  which  can  be  easily  handled  by  2. 

simple  extensions  of  the  basic  ideas.  The  development, 
cost  is  simply  the  SCost/MY  ti.nes  the  area  under  the  Y 

1. 

. (.3945  K)  ‘ • 

(6)  r 

These  are  the  management  parameters  of  the  system  i 

- the  people,  time  and  dollars.  Since  they  are  related 
as  functions  of  time,  we  then  have  cumulative  people 
and  cost  as  well  as  yearly  (or  instantaneous)  people 
_and  cost  at  any  point  during  the  life  cycle. 

In  the  real  world  where  requirements  are  never 
firmly  fixed  and  changes  to  specifications  are  occurr- 
ing at  least  to  som.e  extent,  the  pararieters  K and  t^j 
are  not  completely  fixed  or  deterministic.  There  is 
"noise"  introduced  into  the  system  by  the  continuous 
hum.an  interaction.  Thus,  the  system  has  random  or 
stochastic  components  superimiposed  on  the  deterministic 
behavior.  Sy,  we  are  really  dealing  with  expected 
values  for  y , y and  the  cost  functions  with  some 
"noise"  superimposed. 

Figure  2 illustrates  this  --  the  random  character 
will  not  be  pursued  further  here  except  to  say  it  has 
important  and  practical  uses  in  the  risk  analysis  asso- 
ciated with  initial  operational  capability  and  develop- 
ment cost  (2,4,5). 

Let's  now  shift  our  a'^ention  from  the  empirical 
evidence  to  seme  theoretical  underpinnings  that  will 
shed  more  light  on  the  software  development  process. 

We  will  look  at  the  Rayleigh  Model,  its  parameter 
and  dimentions  and  how  it  appears  to  relate  to  well 
established  physical  principles. 

In  particular  we  will  consider  the  management  ^ 
parameters  K and  tg  so  that  we  may  study  the  imipact  of 
them  on  the  system  behavior. 

Figure  4 shows  the  software  development  process 
as  a transformation.  The  output  quantity  of  source 
statements  is  highly  variable  from  project  to  project. 

This  implies  that  there  are  losses  in  the  process. 

These  losses  would  also  be  highly  variable  if  the 
energy  (work)  relation  is  obeyed 


curve  from  0 to  tg. 
That  is, 

S Cev  * SCost/MY 

= 5c^/my 
S Dev  = 40t  SLC. 


SOFTWARE  DEVELOPMENT  PROCESS 
(TRANSFORMATION  PROCESS) 


WORK  OUT~~l 
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Figure  4.  Software  Development  as  a Transformation. 
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Newton's  Liw  of  Conservation  of  Energy-'work  in  = Work 
Out  = Output  Quantity  of  Sj  + Losses 

Figure  5 shows  the  Rayleigh  Model  in  both  its 
integral  and  derivative  form. 


MANPOWER  UTILIZATION  CURVE 
CUWUlATiVE  MAS^O^CA  UTlllZATiON 
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But  the  ratio  K/t^^  appears  to  have  more  fundamental 

significance.  Its  dimensions  are  those  of  force. 

'when  the  Rayleigh  equation  is  linearized  by 
taking  logarithms 

Log  (yl/t)  » Log  (K/tj^)  <■  (-Log  e)  t^  (10) 

2td^ 

and  then  plotted  as  the  manpower  applied  to  a system 
over  time2  a straight  line  is  produced: 


«0«4tOf  *«>MO(NS  MOOtV 
f !■>••*  * JV» 

II  • f* 


j 


0 2 4 • • 10  12  14  10  1| 


! b 


Figure  5.  ‘ 

The  derivative  equation  is  this: 

r'  * 2kate-at^  (7) 

Where  K is  the  total  life  cycle  manyears  of  the  pro- 
cess and  a is  the  shape  parameter  in  mathematical  terms. 

But  a is  more  fundamental  in  a physical  sense. 

a » 1/2  t^2^  where  tj|  is  the  time  to  reach  peak 
effort.  In  terms  of  software  projects  tj  has  been  em- 
pirically shown  to  correspond  very  closely  to  the  de- 
sign time  (or  the  time  to  reach  initial  operational 
capability)  of  a large  software  project.  Now  we  can 
write  the  Rayleigh  equation  with  the  parameter  tg 
shown  explicitly  by  substituting  for  a; 


y‘=  K/t^  t e-*^^''2td 


Now  it  is  instructive  to  examine  the  dimensions  of  the 
parameters. 

K is  total  work  done  on  the  system, 
tj  and  t are  time  units. 


tj2  X t has  the  dimensions  of  power,  i.e.,  the  time 
rate  of  doing  work  on  the  system,  or  manpower. 


The  cumulative  work  done  on  the  system  is 
Power  X Time  or 

/■  tg  . t 
Y - y'dt 
•'ti  • 0 


/■  tg  . t 
• / / It 

•'ti  • 0 

Y •[*  2Kate 

Jo 

Y • K (1-e  ‘“t*)  MY 


**.;  ■«  IgZ  Ij  * ».  HJMPIS’Ak  i'SHM  >»e*«>in« 

« 'S  iM»  t»sif»*oi'»'CoiTY  e 
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Figure  6. 


When  this  was  done  for  some  40  odd  systems,  it 
was  found  that  the  argument  of  the  intercept,  K/tw^, 
had  a most  interesting  property.  If  the  Number  k/tj^ 
was  small  it  corresponded  with  easy  systems;  if  the 
number  was  large,  it  corresponded  with  hard  systems  and 
appeared  to  follow  a continuum  in  betweqjv  This  sug- 
gested that  the  ratio  K/t^2  represented  di  fficul  ty 
of  a system  in  terms  of  the  programming  effort  to  pro- 
duce it. 

When  this  ratio  was  plotted  against  the  program- 
ing rate  for  some  well  known  projects  like  Safeguard, 
Computer  Systems  Command's  average  programing  rate  and 
typical  figures  like  20,000  statements/MY  for  (easy) 
short-term  commercial  systems  a definite  functional 
pattern  emerged  as  shown  below: 

OiP«<M.rT  >0>« 


Figure  7. 
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This  relationship  is  the  missing  link  in  the  software 
process.  This  can  be  illustrated  by  re-pving  the 
cover  from  the  black  box; 


SOFTWARE  DEVELOPMENT  PROCESS 


inpujG  black  box  transformation  Loyj^f] 

IK.  Id,D  wSk) 


Figure  8.  Black  Box  with  cover  removed. 
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Figure  9.  Feasible  Software  Development  Region. 

A feasible  software  development  region  can  be 
established  semi-intuitively.  Systems  range  in  siae 
(K  MY)  from  1 MY  to  10,000  MY  life  cycle.  Development 
times  (tw)  range  from  a month  or  two  to  five  or  six 
years.  For  large  systems,  the  range  narrows  to  2-5 
years  for  most  systems  of  interest.  Five  years  Is 
limiting  from  an  economic  viewpoint.  Organisations 
can't  afford  to  wait  more  than  five  years  for  a system. 
Two  years  is  the  lov*er  I'mit  because  the  manpower 
buildup  is  too  great.  This  can  be  shown  a number  of 
ways. 


* Addi  tionaT  data  suggests  that  Figure  1 may  be  more 
complex;  i.e.,  it  may  be  a set  of  parallel  lines,  each 
representative  of  a certain  class  of  software  work. 


1.  Brooks  cites  Vysottsky's  observation  that 
large  scale  software  projects  cannot  stand  more  than  a 
30t  per  year  buildup  (presumably  after  the  last  year). 
The  Rayleigh  equation  meeting  this  criterion  is  with 
t,j  > 2 years. 

2.  The  buildup  rate  invokes  Brooks^^)  inter- 
co.;-munication  law.  Complexity  • N (N-1) 

~T~ 

3.  Management  can't  build  and  control  a large 
software  project  at  rates  governed  by  t(j  < 2 years 
without  heroic  measures. 

4.  Der.erally,  internal  rates  of  manpower  genera- 
tion (which  is  a function  of  completion  of  current 
work)  will  not  support  an  extremely  rapid  buildup  on  a 
new  project.  'Jew  hiring  can  offset  this,  but  that 
approach  has  its  own  shortcomings. 

'^/hen  the  feasible  region  is  portrayed  ip  three  di- 
mensions by  introducing  the  difficulty,  K/t^^,  as  the 
third  dimension,  we  have  a difficulty  surface.  Note 
that  as  the  development  time  is  shortened  the  difficul- 
ty increases  dramatically.  Systems  tend  to  fall  on  a 
set  of  lines  which  are  the  trace  of  constant  magnitude 
of  the  difficulty  gradient.  Each  of  these  lines  is 
characteristic  of  the  software  house's  ability  to  do  a 
certain  class  of  work.  The  constants  shown  (8,  15,  27) 
are  preliminary.  Further  work  is  needed  to  refine 
these  and  develop  others. 


Figure  10.  Difficulty  Surface. 

We  can  study  the  rate  of  change  of  difficulty  by 
taking  the  gradient  of  0. 

i1  in  tj  direction 

jj  in  K (and  $ cost)  direction 

grad  D points  almost  completely  in  the  -t<i 
di  rection 

The  importance  of  this  fact  is  that  shortening  the  de- 
velopment time  (by  management  guidance,  say)  dramati- 
cally increases  the  difficulty,  usually  to  the  impossi- 
ble level. 

Plotting  of  all  Computer  Systems  Command  systems  on  the 
Computer  Systems  Command  systems  on  the  difficulty 
field  shows  that  they  fall  on  three  distinct  lines. 
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» 8,  < 15  and  K/t(j^  » 27. 


The  expression  for  grad  0 is: 


grad  0 = *2K  ii  + 1 jj 

t7  7 


We  note  that  the  r-.agnitude  of  the  ii  corponent  has  this 
form  and  the  magnitude  of  grad  0 


/grad  D/ 


(5D/3t)2  + (iO/5K)^ 


= 5 0/3  t * 2K/tj^, since  3 0/3  K « 3 0/3  t (13) 


throughout  the  feasible  region. 


/grad  0/  is  the  magnitude  of  the  maximum  directional 
derivative.  This  points  in  the  negative  ii  (t)  direc- 
tion. The  trace  of  all  such  points  is  a line  K * con- 
stant t^^,  or  K/tj^  ’ ^1'  O'"  change  in 

general ity , 

2K. 


Such  a line  is  the  maximum  difficulty  gradient  line 
that  the  software  organization  is  capable  of  accom- 
plishing. That  is,  as  system  size  increases,  the  de- 
velopment time  will  also  increase  so  as  to  remain  on 
the  trace  of  a normal  to  a constant  gradient  line 
defined  by 


K/t(j'^  = C where  C can  equal  either  8,  15  or  27.* 

(15) 

Study  of  all  Computer  Systems  Command  systems 
shows  that  (1)  if  the  system  is  entirely  new--designed 
and  coded  from  scratch--and  if  the  system  has  many 
interfaces  and  interactions  with  other  systems,  C ’ 8, 
(2)  if  the  system  is  a new  stand-alone  system,  C • 15 
and  (3)  if  the  system  is  a rebuild  or  composite  built 
up  from  existing  systems  where  large  parts  of  the  logic 
and  code  already  exist,  then  C » 27.  (These  values  may 
vary  slightly  from  software  house  to  sofware  house 
depending  upon  the  average  skill  level  of  the  analysts, 
programers  and  managem.ent.  They  are  in  a sense  figures 
of  .merit  or  “learning  curves"  for  a software  house 
doing  certain  classes  of  work.)  The  magnitude  of 
these  constants,  C,  is  instructive. 


K/tj'^  • C i 8 applies  to  new  systems  that  must 


interact  with  others  within  a total  MIS  structure. 

This  is  the  toughest  task  since  no  logic  or  code  exists 
and  the  interfacing  problem  is  formidable.  Accordingly, 
the  difficulty  gradient  for  systems  of  this  type  will 
be  smaller  than  for  other  classes  of  systems. 


K/tj^  • C » 15  applies  to  new  stand-alone  systems. 
These  are  naturally  simpler  because  the  interface 
problem  with  other  systems  is  eliminated. 


k/tj'^  • C • 27  applies  to  rebuilt  systems  or  com- 
posite systems  where  large  segments  of  existing  code 
and  logic  exist.  The  main  task  is  integration,  inter- 
facing and  minor  enhancement.  Here  the  entropy  is 
smaller  because  a significant  portion  of  the  disor- 
deredness has  been  removed  by  prior  work. 


• These  constants  are  preliminary.  It  is  very  likely 
that  6 to  8 others  will  emerge  as  data  becomes  avail- 
able for  different  classes  of  systems. 


The  significance  of  grad  D is  shown  in  the  next 
few  figures.  Note  that  grad  0 is  a vector  which  points 
very  close  to  the  negative  time  direction  throughout 
the  feasible  region. 


It  is  instructive  to  see  how  grad  0 changes  as  a 
result  of  rana.gement  decisions.  We  can  study  this  by 
using  so.me  reasonable  numbers  in  the  expression  for 
grad  0.  The  first  thing  we  observe  is  the  relative  im- 
portance of  the  components. 


Assume  a typical  size  system: 
K = 400  MY  f(i  * 2 years 


grad  0 » i -2(400)  + j 1 * i -800  + j 1 

r3p~  T3T2  ^ T 


< (-30)  t j (.11) 


Importance  of  components 


Magnitude  i_  » 30 

Ratio  J OTTl 


The  i component  (in  the  tine  direction)  is  several 
hundred  times  more  important. 


When  the  manpower  is  cut,  we  get  this  effect: 


Assume  a hypothetical  management  decision  to  cut 
the  life  cycle  cost  of  our  typical  system  by  10«. 


Now,  K = (400-40)  * 360  MY. 


grad  0 » i *2(360)  ^ j 2. 


i (-27) 


j (.11) 


A small  change  of  ■*•3  tip^  change 


3ut  all  in  the  time  direction.  This  says  we 
assume  the  difficulty  is  less  than  it  really  is.  The 
impact  will  be  less  product. 


The  .more  common  case  is  attempted  time  com- 
pression. Assume  that  management  says  instead  "Your 
budget  is  limited  to  400  MY,  but  vve  need  the  system 
in  2.5  years  instead  of  3 years". 


Now  K » 400,  tj  • 2.5  years 


grad  0 » i *2(^00) 

in)3 


i (-51) 


* j (.16) 


The  (difficulty  gradient)  increased  in  size  from 
30  to  51. 


The  result  is  that  shortening  the  natural 
development  time  will  dramatically  increase 
the  system  difficulty. 
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The  risk  analysis  for  project  conpletion  is  pro- 
bably the  'ost  important  aspect  of  any  software  system 
analysis.  This  is  because  the  entire  process  is  ex- 
tre-ely  sensitive  to  small  changes  in  tiiTie.  Why  this 
is  so  was  just  demonstrated  in  terms  of  the  difficulty 
gradient.  Note,  however,  in  the  figure  below  that 
shortening  the  development  time  by  1/2  year  (assuming 
that  we  know  the  job  should  take  3 years)  lowers  the 
probability  of  successful  completion  of  development 
to  the  3;  probability  level.  This  is  practically  im- 
possible and  will  guarantee  a slip.  Brooks  (3)  points 
out  that  adding  more  manpower  will  not  compensate  for 
too  little  time.  Brooks’  statement  is  equivalent  to 
trying  to  manload  the  project  according  to  curve  Y2 
when  it  is  known  that  Y3  is  the  proper  curve. 


PROJECT  COMPLETION 


Figure  11.  Risk  Analysis  for  Project  Completion. 

No  mention  has  been  made  of  how  to  determine  the 
parameters  K and  tj  --  either  for  a new  system  or  for 
a system  already  under  way.  Space  does  not  permit 
this  development  here.  Suffice  it  to  say  that  good 
empirical  methods  have  been  developed  to  do  this  and 
a sound  theoretic  basis  is  well  along  in  development. 
For  information  on  this  aspect  of  the  problem,  the 
reader  is  referred  to  the  references  at  the  end  of  the 
article. 


SUMMARY: 

In  suimiary,  software  systems  follows  a life  cycle 
pattern.  These  systems  are  wel]  described  ^ the 
Rayleigh  (manpower)  equation,  y • 2Kate'*t‘,  and  its 
related  forms.  The  system  has  three  fundamental  para- 
meters: the  life  cycle  effort  (K).  the  development 
time  (tp)  and  the  difficulty  (K/tj^).  Systems  tend 
to  fall  on  a line  normal  to  a constant  difficulty 


gradient.  The  magnitude  of  the  difficulty  gradient  will 
depend  on  the  inherent  entropy  of  the  system.  New  sys- 
tems of  a given  size  that  interact  with  other  systems 
will  have  the  greatest  entropy  and  take  longer  to 
develop.  Rebuilds  of  old  systems,  or  composites  where 
large  portions  of  the  logic  and  rode  have  already  been 
developed  (and  hence  have  reduced  the  entropy)  will 
take  the  least  time  to  develop.  New  stand-alone  sys- 
tems will  fall  in  between. 

The  software  development  region  is  a difficulty 
field.  Each  point  is  uniquely  defined  by  the  three 
Rayleigh  parameters,  K,  t(j,  and  K/tjj‘. 

Systems  seek  their  natural  parameters  and  are  in- 
herently stable.  This  is  along  a line  normal  to  a 
^constant  gradient. 

The  management  implications  of  these  concepts  are: 

Management  cannot  diminish  the  development  time  of 
a system  without  increasing  the  difficulty.  All 
changes  take  place  in  the  negative  time  direction. 
Development  time  is  the  most  sensitive  parameter.  It 
cannot  be  set  arbitrarily  by  management. 

.Given  a proper  set  of  project  parameters,  K,  t^j, 
K/tw*^,  a system  can  be  designed  to  cost  with  only  a 
small  uncertainty. 
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Evolution  Dynamics  - A Phenomenology  of  Software  Maintenance 


M.  M.  Lehman 

Department  of  Computing  and  Control 
Imperial  College  of  Science  and  Technology 
London  S\V7  2BZ 

Abstract.  The  dynamics  of  evolution  of  large  software  systems  has  been  extensively 
studied  in  recent  years.  The  present  paper  briefly  reviews  the  phenomena  observed 
and  the  main  conclusions  that  follow.  The  reader  is  referred  to  the  most  recent  refer- 
ences and  to  other  contributions  to  the  present  Life  Cycle  Management  Workshop  for 
the  data  and  derived  models  on  which  these  conclusions  and  generalizations  are  based. 
T.aken  together,  these  results  constitute  a phenomenology  that  provides  an  essential 
ingredient  of  developing  computer  science. 

Key  Words.  Phenomenology,  Evolution  Dynamics,  Software  Maintenance,  Program- 
ming Process,  Programming  Management,  Life  Cycle  Management. 

1.  Phenomenology 

The  dynamics  of  evolution  of  large  programs  has  been  discussed  in  a number  of  readily 
available  publications,  most  recently  in  (BEL76).  (LEH761,  and  (SLC77).  There  is, 
therefore,  no  reason  to  detail  the  results  or  ihe  data  on  which  they  are  based  once 
again.  The  interested  workshop  participant  can  refer  to  these  papers  and  to  the  earlier 
publications  referenced  in  them.  More  recent  results  extending  the  data  can  be  found 
in  a thesis  by  Chong  (CH077)  and  in  Patterson  (SLC77). 


The  present  contribution  to  the  SLCM  Workshop  is,  therefore,  restricted  to  a restate- 
ment of  the  most  general  conclusions  since  it  is  they  that  suggest  the  existence  of  soft- 
ware phenomenology,  a basis  for  a software  engineering  science.  And  phenomenology 
is  one  view  of  the  theme  of  this  workshop.  We  note  parenthetically  that  analogous 
observations  also  exist  for  other  computer  and  computing-related  activities  leading 
more  generally  to  a concept  of  computing  phenomenology  as  a basis  for  computer  sci- 
ence. 
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The  most  funclamentitl  iziiplication  of  our  observations  is  the  existence  of  deterministic, 
measurable  regularity  in  the  life  cycle  of  an  application  program  or  of  a software  system. 
It  has  been  formalized  in  the  third  of  our  laws  of  evolution  dynamics  (figure  1).  The 
oliservation  contradicts  a basic  assumption  normally  implicitlj’  accepted  by  managers 
of  industrial  and  other  large-scale  human  activity.  These  generallj'  assume  that  the 
progress  and  rate  of  progress  of  a (software)  project  and  the  fate  of  its  product,  (the 
software  system)  is  primarily  the  consequence  of  management  decisions  and  resource 
investment.  Our  conclusion,  based  on  the  measurement  of  a number  of  systems,  proj- 
ects and  organizations,  has  had  to  be  that  in  a software  development  and  maintenance 
environment,  product,  process  and  organizational  parameters  develop  during  the  early 
stages  of  the  life  cycle.  Increasingly  these  then  dominate  the  further  life  of  the  system 
during  the  continuing  maintenance  process,  producing  its  characteristic  features. 

Norden  (SLC77)  and  Putnam  (SLC77)  reach  a similar  conclusion  from  quite  a different 
set  of  observations. 

The  form  and  value  to  which  these  process  descriptors  converge  are  functions  of  many 
factors  relating  to  the  nature  and  history  of  the  product  and  organization.  Precise  deter- 
minants are  not  yet  understood.  But  system  analysis  indicates  that  the  multi-loop  feed- 
back nature  (checks,  balances  and  controls,  for  e.xample)  of  system  development  and 
organizational  management,  is  a major  contributor  to  the  long-term  stability  of  these 
parameters,  probably  in  part  also  of  their  actual  distribution  and  value.  Riordon's 
Studies  (SLC77)  supports  this  by  constructing  continuous  system  models  whose  behavior 
can  be  made  to  reproduce  that  of  various  processes  by  suitable  adjustment  of  their  para- 
meters. 

2.  Evolution 

Our  first  law  asserts  that  large  computer  programs,  software  systems  that  are  used, 
undergo  a continuing  process  of  evolution;  repair,  modification,  enhancement,  adapta- 
tion. The  meaning  of  large  in  this  connection  is  discussed  elsewhere  (BEL77)  as  are 
the  underlying  reasons  for  this  continuous  change  (BEL76),  (LEH76).  The  actuality  of 
the  phenomenon  has  been  confirmed  and  sized  by  measurement  on  some  eight  systems. 
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Law  of  Continuing  Change 


A large  program  that  is  useH  undergoes  continuing  change 
until  it  is  judged  more  cost-effective  to  recreate  it. 


Law  of  Increasing  Entrophy 


The  unstructuredness  (entropy)  of  a program  (system) 
increases  with  age  (change)  unless  work  is  specifically 
invested  to  maintain  or  reduce  it. 


Law  of  Statistically  Smooth  Change 

Measures  of  global  system  and  project  attributes  may 


appear  stochastic  locally  in  time  and  space,  but  are 
cyclically  self-regulating,  with  statistically  identifiable 
invariances  and  well-defined  long  range  trends. 
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It  must  be  stressed  that  this  evolutionary  trend  is  not  primarily  due  to  human  imperfec- 
tion in  identifying  an  application  and  its  needs,  planning  a system  and  implementing  it. 

Nor  is  it  mainly  due  to  the  underdeveloped  nature  of  many  aspects  of  the  programming 
process,  its  technologies  and  its  support  tools;  though  the  absence  of  effective  struc- 
tured specification  methodologies  and  software  interface  control  technologies  undoubtedly 
contributes  significantly  to  the  magnitude  of  change  in  a number  of  ways. 

No'.  Continuing  evolution  is  intrinsic  to  the  very  nature  of  software  and  of  the  use  of 
computing.  It  is  the  visible  sign  of  continuing  interaction  between  the  system  and  its 
environment.  Any  program  is  a model  of  an  activity,  a process  or  an  aspect  of  the 
world  in  which  we  live.  Or  it  may  be  a tool  to  interact  with  and  modify  some  part  of 
that  world.  But  the  world  itself  changes  continuously,  driven  by  a variety  of  natural 
and  human,  sociological,  economic  and  technological  forces.  Hence  the  computer  models, 
the  programs,  must  also  continuously  change.  A program  will  require  maintenance 
even  if  — and  this  rarely  if  ever  occurs  — its  first  implementation  was  perfectly  con- 
ceived, perfectly  designed  and  perfectly  implemented. 

3.  Complexity 

.As  a program  is  changed,  its  comple.xity  will  generally  increase  unless  some  part  of 
the  change  effort  is  specifically  dedicated  to  complexity  control  or  reduction.  Once 
again,  we  cannot  here  repeat  detailed  arguments  or  the  data  that  supports  our  conclu- 
sion (BEL76),  (LEH76).  We  should,  however,  observe  that  the  second  law  may  be  con- 
sidered an  analogue,  perhaps  even  an  instance,  of  the  second  law  of  thermodynamics. 

Belady  (SLC77)  discusses  various  meanings  that  have  been  attached  to  complexity.  He 
reviews  proposals  for  that  view  of  comple-xlty  which  relates  to  software  maintenance 
and  change.  In  that  context,  the  concept  of  complexity  may  be  seen  as  a measure  of 
the  difficulty  of  implementing  a change  totally,  that  Is  correctly  and  completely.  It 
may  be  thought  of  as  reflecting  the  resistance  to  change. 

Note  that  even  this  statement  is  not  unambiguous.  There  are  clearly  many  separate 
concerns;  complexity  of  the  problem  to  be  solved,  of  the  statement  of  requirements, 


of  the  consequent  system  specification,  of  the  procedures  to  be  implemented,  of  the  sub- 
sequent design,  of  the  S3'stem  implementation,  of  the  S3'’stem  in  use,  and  of  the  S3’’stem 
maintenance.  The  implementor's  view  of  the  system  as  he  tries  to  determine  precisely 
what  needs  to  be  done  to  complete  a change,  for  e.xample,  will  be  quite  different  to  that 
of  the  user  who  has  to  understand  how  the  change  will  effect  his  usage  in  terms  of  his 
required  actions  and  the  system's  response. 

Our  observations  have  convinced  us  that  phenomenology  is  a valid  technique  for  software 
process  analysis.  This  suggests  that  despite  the  many  aspects  of  complexity  there  are, 
from  a global  viewpoint,  quantitative  relationships  between  them.  This  is  compatible 
with  our  interpretation  of  the  complexity  of  a system,  as  measuring  the  difficulty  of 
comprehending  the  system  from  the  particular  point  of  one's  interest  or  concern.  Some 
tentative  measures  have  been  suggested  (SLC77),  others  are  being  investigated  (LEH77). 
They  all  relate  in  some  way  to  the  internal  interconnectivity  or  interdependency  of  sys- 
tem elements. 

4.  Regularity 

4. 1 The  Source  of  Data 

In  all  of  the  above  we  have  implied  and  assumed  the  existence  of  a software  phenome- 
nology. This  is  a generalization  which,  if  correct,  provides  the  base  for  a science  of 
software  life-cycle  studies  and  for  the  engineering  and  management  of  software  and  of 
the  software  development,  change  and  adaptation  process  over  the  lifetime  of  an  appli- 
cation or  a system. 

The  data  on  which  our  observations  and  conclusions  are  based,  comes  from  systems 
ranging  in  age  from  3 to  10  years  and  having  been  made  available  to  users  in  from 
ten  to  over  fifty  releases.  They  varied  widely  in  their  application  areas,  functional 
scope,  size,  in  the  sophistication  of  the  related  process,  and  the  implementation  and 
run-time  environments.  Yet  certain  behavioral  features  appear  again  and  again.  It 
appears  justifiable  to  conclude  that  they  represent  basic  properties  of  the  software  life 
process,  but  reflecting  also  aspects  of  the  methodologies  used  and  of  the  Individual 
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organizational  environments.  We  hypothesize  that  many  of  the  characteristics  relate  to 
the  need  for  all  participants  in  the  creation  and  use  of  the  system  to  communicate  with 
one  another,  and  for  the  s.ystem  elements  (subsystems)  to  communicate  with  each  other. 

4.  2 The  First  Law  - Continuing  Change 

We  have  already  discussed  continuing  growth.  In  most  cases,  such  growth  is  at  a declin- 
ing growth  rate.  .And  it  is  possible  to  show  that  the  decline  is  no^due  to  decreased  pres- 
sure for  growth;  that  is  a decrease  in  the  demand  for  additional  capability  to  be  added 
to  the  system.  Nor  is  it  due  to  a withdrawal  of  resources  from  the  maintenance  proc- 
ess. We  can  point  to  many  instances  where  the  growth  rate  declined  whilst  the  people 
applied  and  the  list  of  work  waiting  to  be  done  actually  increased  (BR075).  Thus  the 
decline  is  not  due  to  a positive  management  decision.  It  stems  fi;om  the  increasing 
difficulty  of  working  on  the  system,  the  fact  that  as  the  system  ages  an  ever  greater 
effort  is  required  to  achieve  unit  growth.  It  is  due  to  increasing  comple.xity,  confirmed 
by  the  fact  that  a concerted  effort  directed  to  cleaning  up  a system  and  reducing  its  com- 
ple.xity is  followed  by  a period  during  which  the  growth  rate  reverts  to  its  high  initial  value 
before  settling  back  on  to  the  main  growth  path.  The  Increasing  complexity  has  been 
measured  in  some  of  the  systems  observ’ed,  though  so  far  only  indirectly. 

4.  3 Complexity 

The  source  of  comple.xity  growth  lies  in  the  multifaceted  nature  of  any  change  activity. 

In  particular,  apart  from  the  functional  change  to  be  achieved,  it  may  be  required  to 
complete  the  change  in  the  shortest  possible  time,  svith  minimum  e.xpenditure,  minimum 
performance  degradation,  maximum  performance  improvement  (however  measured), 
minimum  effort  (manpower),  minimum  effort  (Intellectual),  minimum  risk  (errors), 
minimal  structural  change  and  so  on. 

Clearly  these  subgoals  are  likely  to  be  incompatible.  They  cannot  possibly  all  be  simul- 
taneously attained  from  the  point  of  view  of  any  individual  change.  In  terms  of  the 
individual  manager's  interests,  structural  degradation  comes  low  in  the  list  of  priorities  - 
if  it  is  considered  at  all.  Its  debilitating  consequences,  the  benefits  of  attention  to  its 
avoidance  lie  too  far  in  the  future.  And  in  any  case  the  degradation  due  to  any  one  - 
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change  is  likely  to  be  small  — a GOTO  statement  to  provide  access  to  changed  or  addi- 
tional code;  a new  variable  for  temporary  storage;  one  more  access  to  some  global 
variable. 

The  impact  of  structural  decay  lies  in  the  future,  under  commonly  adopted  standards 
it  does  not  affect  the  acceptability  of  an  individual  change.  But  growth  of  system  com- 
plexity is  the  compounded  sum  of  dozens,  even  hundreds  of  "inconsequential"  increases 
in  system-internal  linkages.  Clearly  our  conclusion  must  be  — strictly  control  the 
structural  aspects  of  change  or  periodically  clean  up  and  restructure  or  both. 

4.  4 Cyclicity  and  Long  Range  Trends 

One  of  the  most  striking  features  of  the  data  examined  is  its  tendency  to  regularity. 

In  the  simplest  case,  there  is  a cyclic  variation  about  some  long  range  trend.  That  is 
alternate  values  of  the  system  or  process  measure  — taken  say  at  each  system  release  - 
lie  alternately  above  and  below  a trend  curve.  More  accurately,  one  should  say  that  the 
average  values  of  the  cyclically  varying  parameter  define  or  determine  a long-range 
trend,  in  other  instances,  the  pattern  is  more  complex  but  still  identifiably  regular 
and  defining  a statistically  determinable  trend. 

The  cyclicity  is  interpreted  as  a consequence  of  process  feedbacks.  It  is  the  visible 
evidence  that  the  programming  process  and  its  management  organization  operate  as  a 
feedback-controlled  self-stabilizing  system.  The  trend  curves  and  the  invariances  to 
be  described  reflect  the  long-term  stability  of  these  systems. 

.\s  local  management,  which  is  short-term  oriented,  deviates  from  the  trend,  for 
e.xample  by  increasing  the  rate  of  work  above  that  historically  maintainable,  forces 
develop  that  sooner  or  later  cause  the  rate  to  fall  thereby  maintaining  the  long-range 
trend.  In  the  above  e.xample,  for  instance,  the  excess  work  rate  would  probably  cause 
neglect  of  or  inaccuracy  in  the  documentation.  Equally,  a period  of  excessive  effort  is 
likely  to  cause  an  increased  rate  of  structural  deterioration  and  a larger  number  of 
errors.  Any  one  of  these  factors  is  self-evidently  sufficient  to  subsequently  cause  a 
significant  decline  in  the  work  rate. 
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4.  5 The  Long  Range  Trends 

In  any  large  project  and  large  organization  there  will  be  many  such  related  effects. 

It  is  their  sum  total  over  time  that  determines  the  net  time  pattern,  the  system  life 
cycle.  The  surprising  fact  that  our  investigations  and  those  of  Putnam  (SLC77)  have 
uncovered  is  that  this  net  effect  is  not  stochastic  in  time  reflecting  sequences  of  locally 
inspired,  superfically  unrelated  decisions. 

The  net  effect  as  e.xpressed  by  the  long  range  trends  can  be  identified  by  significant  fits 
to  the  data.  The  resultant  models  can  be,  and  have  been,  used  for  project  and  system 
planning,  committment  and  project  control,  to  produce  for  example,  a (lifetime)  mini- 
iTium  cost  project  or  one  that  meets  its  functional,  time,  and  cost  goals. 

4. 6 Invariance 

Of  all  the  phenomena  measured  and  observed,  that  of  invariance  is  perhaps  the  most 
striking  and  the  most  significant,  particularly  in  view  of  the  fundamental  role  that 
invariances  play  in  modern  physics  for  example.  Several  such  quantities  have  been 
identified.  In  particular,  we  have  observed  that  for  each  of  the  systems  for  which  suf- 
ficient data  has  been  obtained,  the  work  input  or  effort  expended  per  unit-time  has 
remained  constant  over  the  lifetime  of  the  system. 


For  four  of  the  systems  stemming  from  three  organizations,  a count  of  the  number  of 
modules  changed  in  each  release  and  the  length  of  each  release  interval  was  available. 
E.xaming  this  data  as  a function  of  time  (system  age)  — cumulatively  to  eliminate  the 
effects  of  release  overlap  and  slippages  — yields  a trend  that  can  only  be  considered 
linear.  .\n  illustration  is  given  by  Lehman  and  Patterson  (SLC  77,  figure  5). 


This  unexpected  result,  and  its  occurence  in  systems  of  such  widely  different  environ- 
ments, is  as  remarkable  as  it  was  unexpected  — and  unknown  to  the  managements  of 
and  the  participants  in  the  various  activities.  The  constant  rate  of  work  input,  occurred 
in  each  case  despite  advancing  technology,  increasing  tool  support,  changing  language 
and  experience  levels  and,  above  all,  changing  resource  allocation  to  the  activity  — 
both  increasing  and  decreasing.  We  also  point  out  that  this  constant  work  rate  does 
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not  imply  and  did  not  yield  constant  work  output.  Increasing  complexity  already  dis- 
cussed takes  care  of  that.  The  output  per  unit  input  declines  as  the  system  ages. 

The  insensitivity  of  the  work  rate  to  resources  applied,  particularly  manpower,  suggests 
that  many  large  projects  may  be  operating  in  a saturation  mode.  .-Vll  participants  are 
certainly  kept  busy  — keep  each  other  busy  — but  the  system  could  also  have  been  built 
and  maintained  by  a much  smaller  team.  .A  note  of  caution  must  follow.  The  saturated 
mode  of  operation,  once  started,  is  likely  to  have  profound  effects  on  the  nature,  struc- 
ture and  content  of  the  design,  code  and  documentation.  These  remain  reflected  in  the 
system  throughout  its  life  cycle  unless  deliberately  and  consciously  removed.  Thus, 
only  a major  restructuring  or  recreation  effort  can  permit  the  switch  to  a much  smaller 
team. 


5.  The  Phenomenology  of  Management 

The  underlying  cause  of  the  long  term  invariance  of  the  work  input  rate  is  found  in  the 
feedback  nature  of  the  programming  process  as  controlled  by  the  organization  through 
its  management  hierarchy.  In  particular  the  available  data  suggests  that  in  large 
organizations  management  reacts  rather  than  initiates.  Resources  are  applied  to 
methodology  or  tool  development,  for  example,  in  response  to  the  observation  of  a 
declining  production  rate.  .A.fter  all  such  investment  is  basically  anti-regressive 
(LEH74),  it  does  not  contribute  to  immediate  profitability,  to  the  production  of  a mar- 
ketable item. 

That  is,  we  deduce  from  the  data  discussed  that  organizational  response  to  declining 
real  output  or  to  the  experiencing  of  productivity,  quality  and  performance  problems  is 
to  invest  resources  only  to  the  point  where  the  status  quo  has  been  restored.  If  in 
e.xceptional  circumstances  process  development  is  continued  beyond  that  point,  side 
effects  develop  that  prevent  the  full  and  visible  return  on  that  Investment  from  being 
obtained.  The  system  and  organization  has  locked  in,  stabilized  on  a handle  rate  that 
dominates  its  further  development.  Similar  Interpretations  can  be  placed  on  the  other 
phenomena  observed;  that  is,  on  the  various  short  term  cyclcities  and  long  term  trends. 
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Whv  it  should  be  the  handle  or  change  rate  that  is  invariant  is  not  clear  at  this  time. 

Nor  is  it  certain  except  in  outline  why  we  should  observe  absolute  invariance  rather 
than  an  increasing  or  a decreasing  trend.  Thus,  these  first  phenomenological  observa- 
tions must  now  be  followed  by  analysis  and  modeling  efforts  that  will  reveal  the  precise 
structure  and  characteristics  of  the  feedback  systems.  Riordon's  work  (SLC77)  repre- 
sents a significant  step  in  that  direction. 
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. Abstract 


A preliminary  study  of  ALMSA 's  CCSS  logistics  system, 
one  of  the  largest  business  systems  in  the  U.S.,  was  undertaken 
in  order  to  compare  its  evolution  with  that  of  the  systems 
described  by  Be lady  and  Lehman.  To  some  degree  a similarity 
of  behaviour  is  observed  but  there  are  also  significant 
differences.  The  study  is  less  than  two  months  old  so  present 
results  must  be  considered  very  preliminary. 


Introduction 

A reorganization  of  the  U.S.  Army  in  1962  had  as  a 
major  goal  the  maximum  economic  use  of  data  processing  equip- 
ment. This  goal  led  to  the  formation  of  DARCOM’s  Central 
System  Design  Activities  (CSDA's).  The  Automated  Logisitics 
Management  Systems  Activity  (ALMSA),  the  largest  of  the  CSDA's, 
has  designed,  developed  and  implemented  one  of  the  largest 
business  systems  in  the  U.S.  - the  Commodity  Command  Standard 
System  (CCSS). 


iWCKDINO  PAQI  BUI«-NOT  KUBD 


325 


CCSS  is  a nation-wide  data  processing  system  to 
standardise  a part  of  DARCOM ' s (U.S.  Army  Materiel  Development 
and  Readiness  Command,  formerly  Array  Materiel  Command)  logistics 
operation  using  the  IBM  360/65  and  MVT  systems  and  various  peri- 
pherals. CCSS,  which  consists  of  over  1300  modules,  implements 
the  integrated  data  base  concept  and  usually  requires  extensive 
disk  files  on-line.  It  runs  at  some  six  separate  installations 
at  one  of  which,  for  example,  one  file  alone  occupies  eleven 
IBM  3330-type  disks. 

The  CCSS  project  was  initiated  some  ten  years  ago. 

Its  first  release  was  installed  at  the  first  user  site  in  1972. 
During  the  last  years  continuous  pressure  has  been  applied  in 
order  to  expand  new  design  effort.  However,  there  never  seemed 
to  be  enough  resources  to  both  plan  new  design  and  maintain  the 
old.  This  condition  frequently  meant  that  new  design  was 
sacrificed  [7] . 

CCSS  management  and  technical  personnel  have  recently 
had  their  attention  drawn  to  the  Belady-Lehman  studies  [3] . 
Superficially  at  least  it  appeared  that  the  system  and  project 
evolution  behaviour  reported  was  rather  similar  to  that 
experienced  within  ALMSA.  .Moreover  it  was  realised  that  a 
large  amount  cf  data  of  the  type  used  by  Belady  and  Lehman  in 
their  studies  was  in  fact  available  to  describe  CCSS  history. 
Hence  it  was  decided  to  undertake  an  Evolution  Dynamics  study 
of  CCSS,  both  to  compare  experiences  and  to  see  if  planning 
and  management,  tools  similar  to  those  reported,  could  be 
developed. 

This  working  paper  describes  preliminary  observations. 
The  reader  should  remember  that  the  study  is  less  than  two 
months  old. 
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2.  Background 


Since  this  discussion  is  based  on  work  by  Belady, 

Lehman  and  Parr  a brief  summary  is  in  order: 

Observations  on  the  development  of  OS/360  over  its 
many  releases  for  example,  indicate  that  system  size,  com- 
plexity, and  cost  increase  with  time.  The  basic  assumption 
of  programming  evolution  dynamics  is  that  it  is  legitimate, 
indeed  necessary,  to  view  a large  program  and  its  maintenance 
organisation  as  interacting  systems.  Thus  one  must  search 
"for  models  that  represent  laws  that  govern  the  dynamic  be- 
haviour of  the  metasystem  of  organisation,  people,  and  program 
material  involved  in  the  creation  and  maintenance  process"  [3] . 
Field  feedback  is  basic  to  the  process  since  the  system  and 
its  designers  are  being  considered  as  a metasystem.  For  example, 
as  pressure  is  exerted  to  provide  bigger  releases,  the  results 
are  more  complexity,  reduced  quality,  and  other  factors  which 
tend  to  limit  the  grov/th  rate.  Sooner  or  later  this  process 
leads  to  a release  just  for  restructuring/rewriting.  What  is 
also  possible,  and  has  been  observed,  is  an  effect  called  fission 
where  excessive  growth  leads  to  system  breakup. 

The  conclusions  to  date  have  been  summarised  in  three 
'laws  [3]  , [61  . We  are  here  primarily  concerned  with  the  fact 
that  the  evolutionary  pattern  and  project  attributes  of  a large- 
program  organisation  can  be  quantised  and  sized  to  define  cyclic 
variations,  long  term  trends  and  invariaiices.  Once  identified, 
these  natural  system  characteristics  can  be  used  for  planning 
and  management  control.  Identification  of  such  project  and 
system  descriptions  forms  the  first  objective  of  our  present 
study. 

3.  Data  Collection  and  Analysis 

Data  of  the  type  analysed  by  Belady  , Lehman,  Parr 
and  others  has  been  collected  for  CCSS  since  its  first  release. 

In  particular  we  have  available,  though  not  always  for  all 
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releases,  release  numbers  and  dates,  size  in  modules  at  release, 
number  of  modules  effected  by  the  release  (handles  is  the  now 
standard  terminology),  number  of  System  Change  Requests  (SCR's) 
received  and  closed,  number  of  emergency  fixes  processed  and  V 

number  of  Abends  reported.  The  meaning  of  these  various  terms 
should  be  clear  from  the  literature  and  we  do  not  elaborate 
here. 

In  the  first  instance  it  was  necessary  to  establish 
a prima  facie  case  for  or  against  the  legitimacy  of  using  the 
evolution  dynamics  techniques  in  the  CCSS  environment.  This 
has  been  achieved  and  we  now  bring  some  of  the  first  results 
and  preliminary  interpretations,  comparing  them  to  earlier 
published  results  where  relevant. 


4. 


System  Growth 

I 

System  size  has  been  measured  in  modules  and  is  shown 
plotted  as  a function  of  Release  Sequence  Number  (fig.l)  and  of 
age  (fig. 2).  We  observe  two  cycles  of  increasing  growth  rate 
separated  by  a shrinkage  (clean-up)  phenomenon  (at  some  point 
between  releases  with  sequence  numbers  10,14)  as  described  in 
the  references  [3] , [6] . The  two  cycles,  however  stretch  over 
twenty  to  thirty  releases  each  rather  than  the  three  to  six 
described  in  the  references.  This  may  be  related  to  the  fact 
that  until  recently  release  interval  averaged  around  30  days 
for  CCSS  with  net  growth  in  the  range  minus  forty  to  fifty 
modules.  This  must  be  contrasted  with  a release  interval 
range  of  from  three  to  eighteen  months  and  a net  growth  of  from 
minus  fifty  to  about  four  hundred  modules  for  OS/360  in  its 
history  up  to  release  twenty. 


That  is  we  observe  elements  of  the  same  growth  pattern 
between  CCSS  and  the  other  systems,  but  with  differences  that 
we  expect  to  be  able  to  explain  in  terms  of  the  different 
operational,  usage  and  maintenance  environments.  i 
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5.  Net  Growth 


For  OS/360  Belady  and  Leliraan  argue  that  a net  growth 
in  excess  of  some  400  modules  per  release  has  always  led  to 
both  delivery  and  quality  problems.  They  conclude  that  releases 
represent  stability  points  in  continuous  system  evolution. 

Where  excessive  growth  is  permitted  in  a release  interval  how- 
ever, stabilisation  is  not  easily  obtained,  since  the  change 
exceeds  the  intellectual  absorbtive  ability  of  personnel  and 
users.  A similar  conclusion  may  eventually  emerge  from  the 
CCSS  experience  after  correlation  of  figure  2,  with  the  exper- 
iences of  project  participants  and  users.  The  first  impression 
is  certainly  that  a growth  in  excess  of  some  30-40  modules 
leads  to  problems,  shrinkage  and  clean-up.  But  this  observation 
requires  further  investigation. 

6.  Complexity 

In  the  earlier  studies  complexity  measures  were 
approximated  by  the  "Fract ion-of-Modules-Handled"  count.  In 
both  systems  to  which  the  measure  applied  an  increasing  trend 
with  cyclic  variation  was  observed. 

^ In  CCSS  there  is  a striking  difference,  as  in  fig. 4. 

The  fraction-handled  in  fact,  varies  cyclically  about  a constant 
amount  of  about  28%.  A first  attempt  at  explaining  this 
difference  in  behaviour  suggests  that  it  may  be  due  to  a funda- 
mental  architectural  difference  between  the  systems.  For  the 
^ A and  T systems  direct  communication  between  all  modules  of  the 

- system  is  possible  and  permitted  by  protocol.  Thus  as  the  system 

.«  ages  mere  and  more  inter-module  connections  are  established. 

Hence  changes  will,  in  general,  tend  to  spread  increasingly 
across  the  system}  implying,  and  as  implied  by,  the  increasing 
handle  fraction. 

CCSS  consists  of  some  150  applications  programs  each 
comprising  up  to  about  twenty  modules.  Different  programs  can 
communicate  with  or  influence  one  another  only  by  changing 
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values  in  the  files  to  which  they  have  access  or  by  putting 
requests  for  further  action  onto  the  input  queues  of  the  appro- 
priate program  or  programs.  Thus,  with  hindsight,  one  should 
not  expect  the  average  fraction  of  modules  handled  to  change 
significantly  with  age.  That  quantity  is,  in  fact,  not  appro- 
priate as  a complexity  measure  for  a system  of  this  class. 

Clearly  complexity  and  its  relation  to  system  arch- 
itecture and  class  is  a matter  to  be  (and  actively  being)  further 
investigated.  But  even  from  the  very  preliminary  observations, 
important  implications  to  the  system  designer  and  software 
engineer  may  be  drawn. 


7.  Handle  Rate 

The  CCSS  handle-rate  as  a function  of  system  age  is 
shown  in  figures  5 and  6.  The  first  shows  the  cumulative  (to 
eliminate  release  overlap  and  feedback  effects)  number  of 
modules  handled  and  yields  the,  by  now  familiar,  invariance. 

The  average  handle  rate  has  remaind  constant  at  about  8 modules 
per  day  over  a period  of  more  than  five  years.  This  fourth 
instance  of  work-input  rate  invariance  in  fact  leads  us  to 
identify  a fourth  law,  that  of  Constant  Work-Input.  We  shall 
formulate  the  law  more  precisely  in  due  course. 

The  release  by  release  rate  shows  cyclic  variations 
similar  to  those  previously  observed.  Careful  statistical 
analysis  will  be  required  before  any  definitive  conclusions 
can  be  reached. 


8.  Conclusions 

The  results  presented  are  very  preliminary  and  would 
not  now  have  been  published  were  they  not  so  relevant  to  the 
SLCM  workshop.  Nevertheless,  we  have  already  been  able  to 
deduce  some  managerial  guidelines,  for  example,  to  plan  further 
releases  about  the  8 modules  per  day  rate;  to  restrict  growth 
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to  some  30  modules  per  release  or  to  plan  a clean-up  release 
after  any  in  which  a growth  in  excess  of  this  figure  was 
required;  and  to  check  any  release  plan  for  which  the  estimate 
of  modules  to  be  handled  exceeds  some  fraction  of  the  system 
(say  between  20%  and  40%)  as  indicated  by  the  cyclic  pattern. 

Our  investigations  will  continue  and  we  hope  to  report  inter- 
esting and  significant  results  in  due  course. 
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Figure  1: 


System  Size  as  a Function  of  Release  Sequence  Number 
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Figure  2:  System  Size  as  a Function  of  Age 
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Figure  5:  Global  Handle-Rate  Invariance 
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Figure  6:  Handle  Rate  by  Release 
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ABSTRACT 


It  has  been  observed  that  the  life  cycle  of  large  software  systems 
typically  exhibits  several  distinct  phases.  Initially  there  is  a rapid 
increase  in  size;  following  this,  growth  rate  decays,  leading  to  a final 
phase  in  which  the  major  effort  is  directed  towards  maintenance,  with 
little  or  no  further  growth.  The  last  phase  usxially  indicates  end-of-life. 
Based  on  the  concepts  of  evolution  dynamics  postulated  by  Be  lady  and 
Lehman,  the  paper  develops  a fifth-order  nonlinear  differential  equation 
model  of  software  evolution.  The  control  input  takes  the  form  of  a 
cash  flow  which  must  be  allocated  between  three  forms  of  effort:  creation 
of  new  software;  testing  and  fault  correction;  and  maintenance  in  the 
face  of  complexity.  .A  numerical  e.xample  is  presented,  and  a comparison 
is  made  with  e.xisting  growth  data  from  a large  software  system. 
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Introduction 


In  recent  years,  Belady  and  Lehman  [1971,  1972a,  1972b]  have 
collected  and  analysed  data  relating  to  the  development  of  a large 
soft'ware  system.  It  has  been  fo\md  that  measures  of  system  activity 
such  as  number  of  modules  handled,  inter-release  time,  and  total 
number  of  modules  in  the  system,  show  a high  degree  of  regularity 
which  could  not  be  e.xplained  on  the  basis  of  simple  management  decisions. 
The  results  of  this  work  have  been  summarized  by  Lehman  [1974] . The 
basic  premise  of  the  research  is  that  a large  project  of  this  nature 
contains  its  own  dynamics.  Two  postulates  are  of  key  importance  in 
describing  the  system  dynamics: 

(a)  Law  of  increasing  entropy:  "The  entropy  of  a system  increases 
with  time  unless  specific  work  is  executed  to  maintain  or  reduce  it" 
[Lehman  (1974)].  Because  of  changes  in  the  environment  (e.g.,  hardware 
changes,  new  peripherals,  batch/ time- sharing  interaction,  addition  and 
deletion  of  related  software)  work  must  be  done  simply  to  maintain  the 
software  system;  otherwise  it  will  "decay",  and  sooner  or  later  become 
inoperable. 

(b)  Progressive  and  antiregressive  activity:  Progressive  activity, 
in  the  context  of  a software  project,  implies  the  creation  of  new  code. 

The  existence  of  new  code,  according  to  the  law  of  increasing  entropy, 
makes  further  demands  in  the  form  of  antiregressive  activities.  This 
latter  term  includes  fault  correction,  testing  activity,  and  the 
investment  in  methodology  development  to  combat  the  complexity  which 
grows  with  size. 

A further  paper  by  Lehi_an  and  Parr  [1976]  contains  an  important 
generalisation  in  that  it  presents  growth  data  for  two  other  software 
systems.  Each  of  the  three  systems  has  been  developed  independently 
by  different  firms  under  different  conditions  (team  size,  language 
used,  application) . The  features  exhibited  by  the  new  data  show  a 
similar  regularity  which  adds  credence  to  the. precepts  of  evolution 
dynamics. 
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On  the  basis  of  these  postulates,  several  regression  models  have 
been  developed  to  fit  the  observed  data.  In  a more  recent  report 
Belady  and  Lehman  [1975]  have  introduced  a description  which  is  in  a sense 
more  fundamental  than  the  previous  ones.  This  is  a differential 
equation  model  which  attempts  to  explain  the  interaction  of  progression 
and  antiregressive  efforts.  While  simplistic,  it  does  capture  the 
conflicting  resource  demands  of  progressive  and  antiregressive  efforts, 
which  ultimately  limit  the  site  of  the  system.  This  model  is  used  as 
a starting  point  in  section  2 of  the  present  paper.  | 

2.  An  Evolution  Dvnamics  Model 


Feedback  effects  are  inherent  in  the  dynamics  of  many  systems. 

In  analyzing  a software  development  system,  one  might  begin  with  the 

very  simple  model  shown  in  Fig.l.  A certain  level  of  resources  (money,  .! 

manpower,  equipment,  space,  etc.)  is  available  at  a given  time.  Resources 

may  be  allocated  to  the  development  of  new  and  augmented  system  capabilities 

on  the  one  hand,  and  to  fault  correction,  re-design,  and  repair  on  the  other 

e 

hand.  The  first  of  these  is  a progressive  effort,  the  second  antiregressive.  ' 

A dynamic  relationship  arises  because  the  level  of  antiregressive  effort  ^ 

i 

at  any  time  depends  strongly  upon  the  relative  progressive/antiregressive  s 

resources  allocations  in  the  past.  For  instance,  the  sacrifice  of  good  } 

programming  practice  and  proper  documentation  in  the  aid  of  immediate  | 

high  productivity  retxims  later  in  the  form  of  e.xtensive  field  fault  | 

reports  and  user  dissatisfaction.  Lack  of  attention  to  careful  design  | 

resiilts  in  an  inflexible  and  unwieldy  system  which  cannot  be  made  to  I 

grow  as  planned.  I 

With  finite  resources  available,  it  is  evident  that  contention 
exists  between  the  requirements  for  progressive  and  antiregressive 
activities.  What  are  the  long  term  effects  of  a given  allocation  policy? 

What  constitutes  an  optimal  policy?  These  questions,  which  have  yet 
to  be  answered,  are  at  the  heart  of  evolution  d>Tiamics. 
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2 . 1 System  Behaviour 


Before  a general  model  is  considered,  it  is  worth  examining  the 
observed  behaviour  of  an  actual  system.  Fig. 2,  taken  from  Lehman  and 
Parr  [1976]  shows  the  evolution  of  system  site  over  a period  of  some 
ten  years.  To  quote  from  Lehman  and  Parr,  the  system  "is  typified  by 
its  sice,  by  the  richness  of  its  function,  by  the  variety  of  processing 
hardware  devices  and  system  configurations  supported,  by  the  almost 
infinite  variety  of  users  and  application  environments  it  serves,  by  its 
major  use  of  assembly  level  languages,  and  by  the  site  and  geographic 
dispersion  of  development  and  maintenance  teams." 

Two  salient  features  are  apparent  in  Fig. 2.  The  first  is  the 
common  phenomenon  of  a trend  towards  reduced  growth  rate  as  time 
progresses.  The  second  is  more  unusual:  an  oscillation,  beginning 
when  the  system  is  about  1700  days  old,  is  sviperimposed  on  the  trend, 
so  that  the  system  grows  and  shrinks  in  site  with  successive  releases. 
Since  the  latter  is  clearly  not  a planned  phenomenon,  it  is  surmised 
that  such  behaviour  is  associated  with  the  basic  dynamics  of  the 
system  rather  than  the  direct  efforts  of  management. 

It  should  be  noted  that  Lehman  and  Parr  have  collected  a variety 
of  data  relating  to  module  handle  rate,  release  interval,  etc.;  in  this 
paper,  however,  we  shall  confine  our  attention  purely  to  system  growth. 

2.2  Be ladv- Lehman  Model 


Belady  and  Lehman  [1975]  proposed  a model  in  which  activity  is  of 
three  kinds:  progressive  P,  antiregressive  and  additional  work 
related  to  system  complexity,  C.  It  is  hypothesized  that  the  cause  of 
C activity  is  neglect  of  A activity.  The  results  of  cxomulative  neglect 
can  be  removed  only  by  a temporary  increase  in  A.  If  the  total  budget, 

B,  is  limited,  the  result  is  a temporary  decrease  in  progressive  activity  P. 
It  is  assumed  that  B,  P,  A,  and  C can  be  measured  in  cost  per  unit  time. 


Suppose,  in  addition,  that: 


1] 


K = ^ represents  the  inherent  A activity  required 

for  each  unit  of  P activity,  so  that  complexity  does  not  grow, 
m * management  factor,  the  fraction  of  KP  actually  dedicated  by 
management  to  A activity  (0<m<l) . 

A balanced  budget  implies  that  at  any  time 

B » P ^ A + C ...  (1) 

Also  A * mKP,  ...  (2) 

and  it  is  assumed  that  cumulative  neglect  of  antiregressive  activity 
gives  rise  to  cost  C in  the  following  fashion: 

C = /^(l-m)KPdt  ...  (3D 

A block  diagram  of  the  Belady- Lehman  model  is  shown  in  Fig. 3.  The 
open  loop  transfer  function  is 


K (l-mD*(l>Km)s 


where  X = A * C 

s a Laplace  operator 

It  follows  that,  with  initial  conditions  zero,  the  response  to  a step 
of  magnitude  B is 

C(t)  = B[1  - exp(-^  ] . . . (5D 

where  a ^ . . . (6) 

K(l-m) 

The  model,  though  simple,  captures  two  important  aspects  of  the 
dynamics,  namely  resource  sharing  between  progressive  and  antiregressive 
effort  (here  both  A and  C may  be  regarded  as  antiregressive) , and  the 
inevitable  rise  in  cost  of  complexity,  which  finally  absorbs  the  total 
budget  and  limits  further  growth. 
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2.3  .An  Augmented  Linear  Model 


Several  additional  features  are  needed  to  make  the  Belady-Lehman 
model  appro.ximate  reality  more  closely.  In  Fig. 4,  P activity  is  assumed 
to  represent  the  creation  of  new  code,  A activity  the  action  of  fault 
repair,  and  C activity  the  maintenance  of  a complex  system.  Parameter  P 
represents  cash  flow  available  for  new  programming.  The  amount  of  code 
generated  is  fp(P).  where  the  function  f^  may  be  considered  initially  as 
a linear  multiplier  K^.  Code  creation  takes  time,  of  course,  and  it 
is  assumed  that  the  delay  can  be  modelled  as  a first  order  lag  with 
time  constant  years.  System  sice  is  obtained  by  integration  of  the 
resultant  code  production  rate. 

If  the  code  production  rate  at  time  t is  R(t)  (in  modules  per 

month,  for  e.xample) , then,  as  in  the  Belady-Lehman  model,  K R(t)  represents 

the  antiregressive  resources  (in  dollars  per  month)  necessary  to  stem 

the  growth  of  complexity.  In  fact,  the  resources  specified  by  management 

are  f (K  R(t)),  where  f (x)<x.  Moreover,  the  effect  of  this  effort 
m a m “ 

is  delayed,  as  indicated  by  time  constant  t in  Fig. 4.  The  difference 
between  the  required  and  actual  resource  flow  is  shown  in  Fig. 4 as  D. 

The  comple-xity  cost  .arising  from  this  shortfall  is  assumed  to  be  a 
function  the  time  integral  of  D(t),  subject  to  a lag  time  x^.  To 

retain  a linear  model  we  assume  for  the  present  that  functions  and 

f^(.)  are  simply  linear  gains  m and  respectively. 

The  open  loop  transfer  function  of  this  linear  system  is 
given  by 


^ (s) 


KK  [K^(l-m)  * (K^t^  ■►mjs  T^ms^J 


c a 

S(1*T  s)  (It-T  s)  (1  + T s) 

d c p 


Open  loop  poles  occur  at 


...  (7) 


s ■ 0,  , 

T 

a 
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and  zeros  at 


s = _I  C-(K  ym]  + AK  T ^m)-  - drax  K (1-m)] 

■’mr  t <»  C a C C 


...  (8) 


Of  particular  interest  are  the  extreme  cases 
™-n.  Y K K 

^s)  . -i-£-E 

S(1  + T^S)  (1^-TpS) 


...  (9) 


I 


P /•« 


...  (10) 


(1  + T S)  (1+T  S)  (1-^T  S) 

A V P 


When  ni»0,  no  antirejressive  work  is  done,  and  poor  system 

performance  may  be  e.xpected.  This  is  reflected  in  equation  C9)  which 

indicates  that  closed  loop  instability  can  occur  for  sufficiently  large 

values  of  "gain".  A simplified  analysis  with  x^  »x^  ■ x^  »x  shows 

that  the  critical  value  of  K K K , denoted  K,  is  given  by 

a c p - . 

It  can  be  concluded  that  short  time  constants  aid  stability,  a 
result  which  is  intuitively  reasonable. 

Analysis  of  (10)  shows  that  much  great  stability  is  achieved  with  m-1. 
This  is  not  surprising,  as  management  is  in  this  case  doing  all  that  it  can  to 
repair  faults  and  to  reduce  complexity.  .Again,  long  time  constants  have  a 
destabilizing  effect. 

2.4  Nonlinear  Effects 

Since  it  is  amenable  to  analysis,  a linear  model  can  be  valuable 
in  yielding  insight  into  system  performance.  For  several  reasons, 
however,  the  concept  of  linearity  must  be  abandoned  if  a more  accurate 
model  is  desired.  First,  the  system  behaviour  in  Fig. 2 cannot  be 
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reproduced  by  the  model  described  unless  large  step  changes  in 
and  perhaps  m,  occur.  Although  these  parameters  may  vary  with  time, 
there  is  no  reason  to  assume  that  the  variation  takes  a catastrophic 
form. 

Second,  instability  of  the  linear  model  implies  a growing  oscillation 
such  that  system  site,  for  instance,  becomes  negative  over  certain 
periods.  It  is  difficult  to  visioalite  a software  system  of  negative 
magnitude. 

Finally,  there  are  strong  arguments  related  to  the  real  system 

vdiich  point  to  nonlinear  effects.  Let  us  examine  first  the  relationship 

between  the  cash  flow  P for  progressive  work  (coding) , and  the  q\aantity 

of  code  produced.  An  arbitrarily  large  code  production  rate  (we  assume 

that  we  are  dealing  with  meaningful  code)  cannot  be  achieved  even  with 

an  arbitrarily  large  cash  flow.  At  some  point,  saturation  occurs  such 

that  the  introduction  of  additional  manpower  results  in  little  or  no 

effect  on  production  rate  (it  can  be  argued  that  an  actual  decrease  in 

output  may  occur.  Perhaps  so;  this  line  of  thought  has  not  been  pursued, 

however) . As  an  admittedly  crude  approximation  we  can  therefore  assume 

that  f (.)  has  a small  signal  gain  K with  a hard  limiting  characteristic 
P P 

taking  effect  at  output  level  Rp,  as  shown  in  Fig. 5.  Note  that  the  coding 
rate  can  be  negative,  i.e.,  the  system  size  can  decrease  with  time; 
clearly,  this  effect  occurs  in  system  T (Fig.l).  In  the  context  of 
Fig. 4,  a negative  value  of  P indicates  a budget  deficit.  If  the  budget 
B is  fixed,  and  P is  negative,  the  inference  is  that  system  complexity 
has  grown  to  the  point  where  even  the  antiregressive  activities  A and  C 
can  no  longer  be  supported;  there  is  then  no  option  but  to  reduce  the 
total  system  size. 

We  now  turn  our  attention  to  the  complexity  function  fg(.). 

In  the  ideal  case  in  which  programs  are  highly  structured  and  fully 
documented,  it  may  be  argued  that  complexity  cost  varies  linearly  with 
system  size.  In  the  worst  case,  an  exponential  relationship  exists. 

In  this  model  parameter  D represents  a remnant  of  work  undone,  and  the 
worst  case  is  assumed.  Thus,  for  an  input  v,  the  function  f^(v)  is 
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f^Cv)  a a exp  (bv)  ...  (12) 

where  a and  b are  constants. 

A third  nonlinearity  is  in  Fig. 5.  For  negative  input, 

the  multiplier  should  be  zero,  so  that  a "diode  function"  is  created. 

This  effect  was  not  easily  reproduced  by  the  simulation  package 
discussed  in  section  3,  and  had  to  be  ignored. 

A simulation  model  of  evolution  dynamics  is  shown  in  Fig. 6. 

System  equation  are 


X s X 
1 ^2 


*2  ’ 7 


*3  " i 
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X » X 
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...  (13) 
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j 3.  System  Simulation 

i.  j 

3.1  Parameter  Values 

r • ■■  ' " 

I 

I . I 

The  dynamics  of  equation  (13)  were  simulated  on  an  interactive  \ 

1 graphics  system  at  Imperial  College,  London.  Each  "run"  covered  a < 

period  of  ten  years;  a Runge-Kutta  integration  routine  was  used  with  a 
i step  size  of  0.2  years.  Parameter  values  listed  below  are  representative  i 

only,  the  intent  at  this  point  being  to  examine  the  general  behaviour  3 

of  the  model,  rather  than  its  detailed  numerical  validity. 
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r ax  ax  *0.4  vears 
a c p 


Kp  a 100  modules/ me gado liar 


Rp  a 250  modules/year  (saturation  level) 


a a 
b a 0 


0.2  J 


comple.xity  factors 


3.2  Simulation  Results 

Figs. 7-9  show  the  results  associated  with  several  values 
of  input  budget  rate  B and  management  factor  m.  State  variables  plotted 
are: 

Xj^,  the  argument  of  complexity  function 

.x^,  the  code  production  rate  (modules/year) 

x_,  system  size  (modules) 

In  Fig. 7,  the  budget  B is  a step  of  magnitude  $4M  per  year,  and 
m*0.7,  i.e.,  70%  of  the  antiregressive  work  necessary  to  hold  complexity 
in  check  is  in  fact  carried  out.  The  simulation  shows  that  the  code 
production  rate  (the  progressive  element)  increases  to  a maxirntm  of  about 
225  modules/year  at  the  end  of  the  first  year.  At  that  time  the  complexity 
has  increased  to  the  point  where  such  a production  rate  cannot  be  sustained 
within  the  budget  available,  since  an  increasing  resource  demand  is 
being  made  by  A and  C activity.  A balanced  budget  requires  a reduction 
in  P activity,  which  later  leads  to  a reduction  in  A activity.  By  year 
6,  the  system  size  is  close  to  its  asymptotic  size  of  500  modules,  and 
complexity  is  also  nearly  constant.  It  is  evident  that,  with  the  resources 
available,  the  system  has  reached  its  limiting  size. 

Suppose  now  that  it  is  desired  to  increase  both  the  coding  rate 
and  the  final  system  size.  A generous  increase  is  made  in  cash  flow, 
to  $10M  per  year.  The  results  are  shown  in  Fig. 8.  Coding  rate 
rapidly  approaches  its  saturation  level  of  250  modules  per  year,  and 
growth  is  nearly  linear  for  about  2 1/4  years,  the  total  system  size 
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reaching  490  modules  at  that  point.  Complexity  has  also  risen  rapidly. 
Shortly  after  time  T , the  resource  conflict  assumes  the  form  of  a 

K 

crisis.  Prior  to  Tj^,  the  system  had  excess  manpower  (indicated  by 
saturation  in  the  coding  rate) ; now  manpower  is  shifted  suddenly 
away  from  P towards  C activity.  Within  a further  six  months  growth  is 
at  a standstill,  and  a budget  deficit  is  incurred.  To  reduce  complexity 
cost,  it  is  then  necessary  to  reduce  system  size,  and  an  oscillatory 
pattern  sets  in.  Effort  shifts  back  and  forth  between  progressive  and 
antiregressive  (both  A and  C)  activity.  Each  cycle  is  characterised 
by  a rapid  initial  growth  whose  resulting  comple.xity  cannot  be  handled 
within  u.ie  resoxirces  available;  a partial  collapse  ensues,  complexity 
diminishes,  and  the  cycle  repeats.  In  this  case  a mean  system  size  of 
680  modules  is  reached  by  year  10,  with  a superimposed  oscillation  of 
magnitude  ^40  modules. 

In  Fig. 9 the  exponential  complexity  factor  b is  taken  as  0.14, 
rather  than  0.2  as  previously.  This  corresponds  to  the  e.xpenditure  of 
additional  effort  in  maintaining  well  strcutured  software,  so  that  the 
cost  of  increasing  complexity  is  less  severe  than  previously.  Again, 
howejver,  the  system  budget  is  SlOM  per  year.  Fig. 9 shows,  however,  that 
the  system  now  continues  to  grow  "gracefully”  over  the  ten  year  period 
with  an  asymptotic  value  of  about  1400  modules. 

4.  Conclusion 

A nonlinear  differential  equation  model  of  evolution  dynamics 
has  been  developed,  and  preliminary  analysis  and  simulation  have  been 
carried  out.  It  has  been  found  that  the  dynamic  model  is  capable  of 
reproducing  some  important  phenomena  observed  in  the  data  and  that,  in 
turn,  model  parameters  can  be  related  to  observed  characteristics  of  the 
actual  system. 

While  the  results  are  promising,  a great  deal  of  work  must  be 
done  before  practical  results  ( in  the  form  of  an  accurate  predictive 
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model)  can  be  achieved.  The  first  step  is  the  determination  of  a 
suitable  model  structure.  The  one  presented  here  is  oversimplified, 
in  that  no  account  is  taken  of  important  variables  such  as  handle  rate, 
inter-release  time,  and  fault  rate.  In  addition,  it  is  incomplete  in 
that  it  contains  no  complexity  component  which  is  purely  a function  of 
system  size.  It  is  therefore  possible,  with  the  present  model,  to  set 
m»l  and  achieve  indefinite  growth  with  negligible  complexity  cost,  a 
result  unlikely  to  be  duplicated  in  the  real  world.  Indeed,  the  question 
of  the  structure  of  a complexity  model  alone  is  one  worthy  of  intensive 
effort. 

The  second  step  is  the  determination  of  realistic  system  paramter 
values.  The  present  data  is  inadequate  to  use  for  any  form  of  statistical 
parameter  estimation.  It  is  possible  that  more  detailed  estimates  of 
time  constants,  for  instance,  can  be  obtained  by  the  collection  of  further 
data.  Clearly  the  structure  of  the  model  developed  will  have  a strong 
influence  on  data  requirements  generally. 

The  third  major  stage  is  the  application  of  control  theory  to 
the  models  developed  earlier.  Providing  these  are  realistic,  the 
"payoff”  here  can  be  considerable  in  financial  terms.  Consider  as  an 
example  the  system  growth  curves  of  Fig. 7-9.  In  Fig. 7,  some  300  modules 
were  developed  and  maintained  over  a ten  year  period  at  an  average 
cost  of  $80,000  per  module  (recall  that  the  budget  was  $4M  per  year 
for  ten  years).  In  Fig. 3 a mean  level  of  680  modules  was  achieved 
at  an  average  cost  of  $147,000  per  module.  It  seems  evident  that  great 
savings  can  be  made  by  the  correct  control  strategy.  Even  if  such  a 


strategy  cannot  be  applied  in  detail  (due  to  stochastic  perturbations 
such..as  sudden  demands  of  major  customers),  the  results  should  provide 
valuable  guidelines  to  management. 

An  additional  line  of  research,  which  is  probably  premature  at 
present,  is  the  application  of  evolution  dynamics  principles  to  a wider  group 
of  coanerical,  indvistrial  and  socio-economic  systems.  i; 

n 
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The  work  described  in  this  paper  constitutes  a bare  beginning 
of  a serious  effort  to  apply  feedback  modelling  techniques  to  evolution 
dynamics.  However,  the  results  to  date  indicate  quite  clearly  that  the 
effort  is  likely  to  prove  rewarding  in  both  academic  and  industrial  terms. 
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THE  DYNAMICS  OF  SOFTWARE  DEVELOPMENT  I 


ABSTRACI 


GEORGE  J.  FIX 


The  purpose  of  this  study  is  to  develop  a macro-methodology  for 


management  that  will  provide  accurate  estimates  of  manpower  and  time  to  reach 
critical  milestones  of  software  projects.  There  are  two  basic  aspects  of 
this  subject,  namely, 

(1)  System  dynamics  - a model  that  will  describe  the  evolution  of  the 
project  in  time. 

(2)  Equation  of  state  for  the  project  - the  fundamental  relationships 
between  input  parameters  to  the  dynamic  model.  This  report  treats  the  first 
subject  by  developing  a differential  equation  of  the  process  which  allows  one 
to  model  the  effects  of  changes  in  requirements  in  the  middle  of  a projects' 
development.  From  the  project  control  viewpoint  this  permits  one  to 
quantitatively  determine  the  cost  impact  of  changes  to  an  ongoing  development 
and  assess  its  economic  value  before  implementation. 


THE  DYNAMICS  OF  SOFTWARE  DEVELOPMENT  I 


G.  J.  FIX 


1.  INTRODUCTION.  Application  software  development  has  been  an  ) 

area  where  standard  managerial  and  cost  controls  have  proven  ineffective 
in  many  important  respects.  Brooks  [1]  and  Putnam  [2]  have  pointed  out 
instances  of  actual  costs  several  times  the  initial  budgeted  cost  and 
time  to  initial  operational  capability  sometimes  twice  as  long  as 
planned. 

The  purpose  of  this  study  is  to  develop  a macro-methodology  J 

for  management  that  will  provide  accurate  estimates  of  manpower,  and 
time  to  reach  critical  milestones  of  software  projects.  There  are  two  ( 
basic  aspects  of  this  subject,  namely 

(1)  System  dynamics  - a model  that  will  describe  the 
evolution  of  the  project  in  time 

(2)  Equation  of  state  for  project  - the  fundamental  relation- 
ships between  input  parameters  to  the  dynamic  model.  ^ 

In  this  report  we  shall  treat  only  the  first  subject  leaving  the 
second  to  a latter  report.  Thu  starting  point  is  the  software  model 
developed  by  Putnam  £2]  which  was  inspired  in  part  by  the  work  of 
Norden  £3].  We  shall  show  in  Section  2 that' this  model  is  equivalent 
to  a differential  equation  involving  time  and  man-year  requirements.  This 
equation  is  not  the  entire  story  since  it  contains  parameters  which  must 
be  supplied  through  the  project's  equation  of  state.  However,  the  differ- 
ential equation  does  offer  an  improvement  over  the  "Norden  equation"  of  £3Q 
in  that  it  covers  cases  not  treated  by  the  latter.  For  example,  it  can  be 
used  to  model  the  effects  of  a change  in  requirements  in  the  middle  of  the 
project's  development.  These  and  other  examples  are  given  in  Section  3. 

The  last  section  contains  a description  of  the  computer  program  used  in 
these  simulations. 


2.  THE  DYN^IC  MODEL.  Putnam  [2]  has  identified  two  basic  para- 
meters in  any  software  project,  namely  the  time  t^  to  peak  development  and 
the  system  difficulty  D.  The  total  number  oTTeoui red  man-years  k ^s  related 
to  these  parameters  by 


parameters  by  . 

D-K/t5  (1) 

In  a "Norden  type  evolution",  Putnam  has  shown  that  number  of  man-years 
any  given  time  is  given  by 

Y(t)*  K (l-expC-p(t;td)t])  (2) 

where  p is  the  learning  coefficient.  Putnam  has  suggested  that 
p(t;td)-  (3) 

describes  software  projects  quite  adequately.  Observe  that  implicit  in 
the  choice  is  an  assumption  that  the  system  learns  (or  better  Improves 


at 


I. 


Its  ability  to  overcome  problems  in  coding,  design,  etc.)  in  a linear 
way  witfv  time,  i .e. , 

p(tjt^)  “ t. 

That  the  coefficient  p has  the  specific  form  (3)  follows  from  the 
definition  of  t<j  which  we  recall  is  the  time  where 

y(td)=‘0.  , 

Putnam  has  shown  that  the  S shaped  "Rayleigh  curve"  (2)  provides 
an  excellent  description  of  the  time  evolution  of  y under  ideal  conditions. 
It  has  in  addition  the  attractive  feature  of  a superposition  of  similar 
S shaped  curves  representing  coding,  design,  etc.  See  Putnam  [2]. 

The  relation  (2)  does  have  one  major  disadvantage  that  limits 
its  usefulness  for  some  software  projects,  namely  it  assumes  that  the 
systen  difficulty  0 is  constant  throughout.  This,  for  example,  does  not 
permit  the  model  to  be  used  for  projects  where  objectives  or  software 
requirements  are  changed  before  the  completion  of  the  project.  In  the 
remaining  part  of  this  section  we  shall  develop  an  alternate  model  which 
reduces  to  (2)  when  0 is  constant,  but  can  also  be  used  when  0 is  variable. 


The  idea  is  to  view  software  development  as  a dynamic  system 
tending  to  a steady  state,  i.e., 

y(t)  K»y(<»  ) as  t-*  • , (4) 

There  are  two  types  of  forces  acting  on  the  system,  one  being  negative 
(entropy  to  be  overcome  for  completion  of  the  project)  and  the  other  is 
positive.  The  negative  force  has  the  form 

0 - y(t)  (5) 

and  is  the  effective  system  difficulty  at  time  t.  As  above  D is  the  total 
system  difficulty.  It  is  determined  from  the  projects  eouation  of  state, 
and  we  admit  the  possibility  that  it  may  change  in  time  (as  it  will  if  for 
example  the  system's  requirements  are  changed  in  the  midst  of  the  project). 
Me  also  admit  the  dependence  of  0 on  y,  y,  i.e., 

D-0(t,y,^), 

which  reflects  Brooks'  Law  [1],  namely  that  the  system  difficulty  will 
depend  on  intercommunication  and  hence  increase  with  y,  or  say  y. 


The  positive  force  works  against  (5)  and  reflects  the  ability  to 
learn  that  is  fundamental  to  Norden's  model.  In  particular  it  has  the- 
form 

p(t;t(j)^.  (6) 

and  increases  with  the  learning  coefficient  p.  A balance  of  forces 
gives 


d^y  . - p(t,t<i)  ^ D - 


or  what  is  the  same, 


et- 

+ p(t,t<i)  d^  » I » D 


(7) 
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with  the  initial  conditions. 


(8) 


y(0)  « ^(0)  - 0 
dt 

If  we  use  the  linear  learning  law  p'  t,  then  (7)  becomes 


dy  + 1 y - D 
dt  tj 


(9) 


Observe  that  if  D is  a function  only  of  time  with 

0(  «^)  » Tim  0(t)  . 

t-*-  • - 

then  the  solution  y(t)  of  C8)“(9)  will  satisfy  (4)  with  K»D(«)  tf.  Indeed, 
using  variations  of  parameters  we  have 

Y(t)  « t^  / expf-^^  (t/tji)2- Ig  (t/t(j)^ / ds  D(s)  (10) 

O ' 0 


The  relation  (4)  following  by  integrating  by  parts  and  then  passing  to  the 
limit  as  t-*-*.  Also  observe  that  if  0 is  a constant,  then 

z t 

Y(t)  - td  0 / expr+  li  (t  )2  - (t)2jll  dr  (11) 

0 td  td  " td 

» K 5j  - exp  C-  ^ (t/tjj)2]  V (12) 

which  is  exactly  Norden's  equation. 


3.  SOME  EXAMPLES.  In  this  section  we  report  examples  that  illustrate 
the  dynamic  model  [9,  Section  2].  In  each  case  the  system  difficulty  D is 
a function  only  of  time,  and  hence  [(10),  Section  2]  is  an  exact  solution, 
nevertheless  it  was  more  convenient  to  use  a numerical  solution  of  (9).  The 
program  is  described  in  the  next  section,  and  a listing  is  included  for  future 
reference. 


Two  systems  were  chosen  for  purposes  of  illustration,  one  large  and 
the  other  medium  sized.  The  data  came  from  IBM  and  was  communicated  to  the 
author  by  Putnam. 


and 


The  first  example  is  IBM  program  No.  30  which  has 
, D ■ 77.73  man-years 


td  ■ 2.125  years 

The  Norden  curves  for  Y and  y are  shown  in  Figures  3-1  and  3-2,  respectively, 
^jso  plotted  in  these  figures  is  the  dynamic  model  [9,  Section  2]  where  at 

the  was  increased  by  a factor  of  1.25.  Observe  that  this  resulted 

in  an  increased  value  of  K (from  roughly  350  to  430)  and  anincrease  in 
td  (from  2.125  to  2.4).  Observe  also  that  the  effect  on  the  system  due  to 
the  sudden  change  in  D was  not  felt  instantaneously.  The  greatest  effects 
were  seen  roughly  a year  after  the  impluse.  This  shows  that  even  a modest 
perturbation  of  the  system  can  have  definite  effects  on  large  systems,  and 
that  these  effects  may  not  be  apparent  until  a much  later  time. 
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where 


A similar  experiment  was  done  on  the  smaller  IBM  program  No.  5 
K « 25.75 


1 


tl 

1 


T • 


i 


and 

tjj  » .83. 

Observe  that  the  effects  of  the  perturbation  are  less  severe  and  detect! ble 
sooner. 

4.  Numerical  Scheme.  We  rewrite  the  differential  aquation  [9,  Section  2] 
as  a first  order  system 

y-v  (1) 

V » -td2(Y  + tv)  + 0(t,y,v)  (2) 


To  integrate  this  equation  we  introduct  a grid 


with 

0 ■ tg.  < t < . . . 

V 

V 

tk  + ^ (tk 

tk  +1). 

Put 

Then 

the  approximation  is 

(3) 

C V % • Yk  + 4t  v,j 

2 „2 

and 

'•  “ '^k  “ 4t  tft 

. h ''k)  ^ D(t,^.Y,^.v^) 

4 

(4) 

j ^k  + 1 * ^k  + at  V[^  + 

(5) 

1 vk  + 1 * vk  - at  td^  (Y,^ 

+ ^k  + ^ Vk  + 

(6) 

+ at  D Ctk  + 

h f ^k  + ^ , V|^  + 

Where  at  » tj^  + •]  - t^.  This  is  a 2nd  order  predi ctor-corrector  method,  and  is 
of  the  Runga-Kutta  type. 

The  functions  C and  XK  are  the  coefficients  of  y,Y,  (respectively) 
and  FCN  is  the  time  dependent  systen  difficulty. 
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// 


1 _C_  _ 

PR3GRAM  SQFTOYNi  input, OUTPUT^  PUNCH,TA065-INPUT,  TAPE6-aUTPUT>TAi.:7 

■ •■PUNCH)  - 

^■C~_0YNAMIC_NQ06L  FOR  SOFTWARE  DEVELOPMENT 

5 c _■  _ BASIC  variables  

C "■  Ul-MAN-YEARS  AT  CURRENT  TIME  LEVEL  '1 

■ C * U2-RATE  OF  CHANGE  CF  Y 
C _■  O-SYSTEMOIFFICULITY 
C _ ^ K-MAX  VALUE  OF  Y IN  NQROENS  MODEL 
10  _C  TD-TIME  TO  PEAK  DEVELOPMENT  IN  NOROENS  “ODFL 

C " C(r,OT>Ul,U2)-LEARING  COEFFICIENT  AT  TI'<E  {I-1)*DT 

C XK(I,0T>U1,U2)*£NTR0PY  COEFFICIENT  AT  TI»*E  (I-l)*OT i 

' C'  _ ~ 

CdMMON/PARMfR/0,TD 

15 0lMENSlbN_Ul(5000),U2{5000)  _ 

;; _ _ 

C INPUT  Oa'TA 

C 

REA0(5#1000)  D 

20  _ _ PEA0(5,i000)  TO  _ 

1000_  FORMATCEIS.T) 

■■WRITE{6»1001)  0,TD 

100 1 FORMAT (»  SYSTEM  0IFFICULTY-*,E15.7,2X, •0£VEL0PMENT_TIME-*,E15.7) 

C' 

25  C READ  0ATA_FOR  NUMERICAL  INTEGRATI0N_QF  DYNAMIC  ECUATIONS 

^ C ^ 

R£AD{5,1002)  NTS  _ _ 

1002  "FORMAT!  I^)  __  _ , 

' 1 RSA0(5,1C00)  DT  " 1 _ 

30_  _ _ 0Tm»0T**5 

NTaT»2*NTS 

^ 

C START  OF  TIME  EVOLUTION  _ 

c_'  ' ■ _____ 

35  _c initial'  data 

I"_  _ UKD-O.  _ _ _ 

^ U2(l)fO.  _ 

__  WRrTE(6»l0d3)  __  

1003_  F0RMAT(5X,»  TIM£*,lAX,*Y*,li)X,^*DY*) 

40"_' _T»3,  

DO  10  IT»l,NTaT»2  ■ 


T«T^DT 


PROGRAM  SQFTDYN 


.7A/7A  OPT-0  TRACc  Ct3UG 


PTN  A*6*A28 


<^5 


50 


55 


60 


65 


ITl-IT+1  

UlM-UKIT) 

U2M-U2(ITJ 

U1Z"U1M+CTH*U2M  _ 

LKITl  )«U1Z 

CC-C ( ITl»0Trt,Ul/U2) 

XKK-XK(ITU0TH»U1#U2) 

_.FE?FCN{ITl,OTHf  U1»U2) 

FF»FE-CC»U2rt-XKK*UlM 

U2Z-U2M+LTH*FF  __  

lj2(in)«U2Z  ~ ' L _ 
rT2*rTi-*>i  __ 
U1?»UIM+DT*U2Z 
Ul(.rT2)-UlP  


1004 

10 


CC«CIIT2>0TH»U1#U2) 

XKK-XK  CIT2>0TH,U1>U2)  __ 

_ FE-FCN(IT2»0TH#UI»U2) 
FF-Fc-CC*U2Z-XKK»UlZ 
U2?-U2«+0T*FF 
li2(IT2)«U2P 

WRITE(6» 1004)  T»U1P»U2P 
FaRMAr(E15*7^2X>E15*7/2X#£15*7) 

CONTINUE  

STOP  _ ' 

ENC 


function  C(I»0LT»U1>U2) 

CnMNON/PARNTR/OtTO 

DIMENSION  01(5000), 02(5000) 

T-FLOATd-l)  ♦OLT_ 

C»r/(T0**2) 

^ RETURN  ■ 

END 


_FUNCTION  XK(I,DLT,U1»U2) 
C0MM0N/PARMTR/0»T0 
DIMENSION  U1(S000),U2(5000) 
■ T-FLQATd-D^OLT 
■xk«(1./T0)**2 
RETURN  

6N0  ■'  ''  __  .... 


1 ) 

i 


ii 
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SOFTWARE  COMPLEXITY 


L.  A.  Belady 

IBM  T.J. Watson  Research  Center 
Yorktown  Heights,  N.Y.  10598 

August  1977 


ABSTRACT:  Earlier  work  on  program  complexity  is  briefly 

reviewed.  Observed  large  scale  software  evolution  and  some 
measures  of  its  complexity  are  then  described.  An 
information  theoretical  model  by  Beilner  and  Belady 
formulated  but  unpublished  In  1975"  is  introduced  to 
characterize  the  three  distinguishable  components  of 
software  modification  effort:  identification  of  the 
impacted  modules,  coordination  of  changes  among  the  modules, 
and  finally  the  implementation  of  changes.  Some  results  are 
then  related  to  software  design,  maintenance  and 
documentation.  (This  paper  was  prepared  for  presentation  at 
the  Workshop  on  Software  Life  Cycle  Management,  sponsored  by 
the  U.S  Army  Computer  Systems  Command,  August  22-23,  1977.) 
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INTRODUCTION 


It  is  a natural  desire  to  compactly  capture  and  attempt  to 
measure  the  agony  we  encounter  while  working  with  software, 
spanning  the  entire  life  cycle  from  requirements  to 
operation  and  maintenance.  The  program  in  its  machinable 
form  is  the  most  obvious  manifestation  of  the  reasons  for 
great  and  often  unpredictable  cost.  Thus,  not  surprisingly, 
the  majority  of  measures  proposed  so  far  simply  summarize 
some  countable  parameters  of  this  end  product,  in  the  hope 
of  being  able  to  characterize  and  analyze  difficulties  of 
creation,  operation  and  modification  of  software  <GIL>  or  at 
least  reveal  how  frequently  different  software  constructs 
are  applied  In  actual  use  <ELS>,  <KNU>. 

The  study  of  computational  complexity  <AH0>,  for  example 
seeks  to  find  for  a given  algorithm  the  cost  - processing 
time  or  m.emory  space  - as  a function  of  problem  size,  but 
says  little  about  other  properties.  A relatively  new  and 
related  area  of  research  is  algorithmic  information  theory 
<CHA>.  Here  for  instance  one  wishes  to  establish  the 
minimum  amount  of  information  necessary  to  specify  an 
algorithm.  Another  sought  after  measure  is  the  probability 
that  a random  bit  pattern  acting  as  a program  produces  a 
particular  output  and  then  terminates. 

Other  work  is  concerned  with  identifying  descriptive 
parameters,  essentially  those  which  describe  the  control 
structure  of  a finished  program  <BAS>,  <MCC>,  <HAL>. 
Typically,  the  number  of  branching  points  is  considered 
dominant  while  the  contribution  of  straight  line  code  to 
complexity  is  often  suppressed.  Occasionally  the  formula 
for  entropy  is  applied,  but  rarely  explored  to  its 
probabilistic  foundation. 

It  seems  to  be  di scouragingl y difficult  to  construct  a 
complexity  index  which  measures  not  only  the  end  product  but 
also  the  process  leading  to  It.  And  since  it  is  becoming 
increasingly  evident  that  the  cost  of  changing  existing 
programs  prevails  over  that  of  their  creation,  perhaps 
quantized  phenomena  which  measure  dl f f i cul ty  and 
consequences  of  program  modification  bring  us  a bit  closer 
to  a satisfactory  metric. 

Three  approaches  to  capture  modification  complexity  are 
reviewed  in  this  paper.  The  first  one  is  essentially  a 
measure  of  diffusion  and  stratification  of  changes  into  the 
rest  of  the  system  as  observed  in  connection  with  actual 
large  software  and  later  reported  <BEL>.  The  second  one 
focuses  on  the  relative  locality  of  declared  variables  for 
software  expressed  in  block  oriented  languages  <H00>. 
Finally,  a probabilistic  model  of  programming  complexity  is 
offered,  leading  to  the  formulation  of  an  entropy  measure. 
Some  Ideas  of  information  theory  are  then  interpreted  in 
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this  context  and  conclusions  drawn  related  to  design, 
documentation  and  maintenance  practices. 

COMPLEXITY  GROWTH  OBSERVED  - 

DIFFUSION  OF  CHANGES  AND  EXPANSION  OF  SCOPE 

In  this  paper  we  consider  all  software  systems  made  of 
interdependent  modules,  not  simply  a collection.  The 
internal  detail  of  these  modules  is  hidden  and  will  not 
concern  us  here.  We  Just  observe,  and  attempt  to 
characterize,  the  response  of  this  system  of  modules  to  a 
series  of  changes  (error  repair,  functional  enhancement).  A 
module  is  subjected  to  a change  if  its  machinable  forms 
before  and  after  the  change  are  not  identical.  Other 
characteristics  of  a change  are  not  examined  here. 

Large  systems  typically  evolve  over  a series  of  so  called 
releases.  At  each  release  batches  of  changes,  accumulating 
over  a period  of,  say,  several  months,  are  applied. 

Examples  of  this  are  operating  systems  and  large  application 
software,  such  as  electronic  line  switching,  banking,  etc. 

Let  h be  the  ratio  of  changed  modules  to  the  total  number  of 
modules  before  the  change.  On  a large  operating  system, 
which  had  been  evolving  over  a decade  of  about  twenty 
releases,  h was  observed  <8EL>  to  be  monoton i cal  1 y 
Increasing  and  approaching  unity  as  depicted  in  the  figure 
below: 


Interestingly,  the  rate  of  modification  activity  over  the 
entire  period  of  observation  was  verified  to  be  essentially 
steady  state,  yet  the  dispersion  of  changes  displays  a 
faster  than  linear  tendency,  though  obviously  bounded  by 
unity.  The  spread  of  changes  is  thus  greater  in  an  old  and 
evidently  more  unstructured  system  - an  indication  of 
structural  aging. 

From  among  many  potential  system  parameters  the  authors 
found  h to  be  the  closest  to  an  acceptable  index  of 
complexity.  If  is  an  observed,  and  directly  measurable, 
quantity  which  also  describes  a system  property  , namely 
its  resistance  to  change.  Indeed,  effort  spent  is  likely  to 
be  proportional  to  the  number  of  modules  involved  in  system 
mod i f i cat i on  . 

The  same  report  <BEL>  also  offers  a model  of  error  repair 
diffusion.  Following  this,  during  a repair  interval  between 
two  releases  some  errors  are  removed,  with  the  net  result  of 
leaving  residual  errors  and  injecting  new  ones  due  to 
imperfection  of  the  repair  process  itself.  This  error  flow 
recursively  applied  to  each  interrelease  interval  displays 
then  an  exponential  growth  of  the  number  of  error  classes, 
each  class  with  different  process  history.  This 
stratification  is  offered  as  a major  contributor  to  the 
deterioration  of  structure,  hence  system  complexity. 

The  second  approach  mentioned  in  the  introduction  has  its 
origin  in  studies  performed  on  a quite  differenct  software 
system  which  was  written  in  Algol  60,  a block  oriented 
language  <HCC>.  In  these  studies  another  phenomenon  was 
dicovered  as  a potential  candidate  for  a novel  complexity 
measure,  as  explained  below. 

Due  to  explicit  declaration  of  all  referenced  variables,  it 
was  possible  to  count  them  all,  and  subsequently  group  them 
according  to  the  block  at  which  each  was  declared.  This 
spans  a distribution,  so  one  can,  for  example,  define  b,  the 
expected  block  level  or  scope  of  variables  11:  was  noticed 
that  over  a period  of  system  evolution  the  expected  scope 
was  monoton i cal  1 y increasing,  analogous  to  the  growing 
unstructuredness  of  the  operating  system  described  above. 

The  following  figure  depicts  this  phenomenon. 
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One  of  the  goals  of  structured  design  is  to  keep  variables 
as  local  as  possible  in  order  to  control  complexity. 

However,  this  becomes  increasingly  difficult  when  changes 
are  to  be  effected  which  do  not  fit  into  the  original 
architecture.  Structural  clarity  unavoidably  suffers. 
Perhaps  b and  its  variation  during  system  evolution  are 
worth  while  monitoring  as  complexity  indicators  during  the 
entire  life  cycle  of  large  software,  whenever  disciplined 
declaration  of  variables  is  possible. 

A PROBABILISTIC  COMPLEXITY  MODEL  - 
INFORMATION  THEORY  APPLIED 

The  previous  section  already  Implied  the  two  major  aspects 
of  software  modification  complexity:  complexity  contributed 
by  the  content  of  the  change  (process)  and  that  by  the 
internal  structure  (system).  In  building  our  model  we  wish 
to  keep  these  two  contributors  to  complexity  as  separate  as 
possible.  More  formally: 

- Let  the  system  under  consideration  consist  of 
n el  ements  or.  modules 


1 


~ The  system  evolution  is  decomposable  into  a series 
of  next  change  requests. 

- Each  next  change  generates  the  following  three 
mod i f i cat i on  activities/  or  phases  , and  in  this 
order: 

- Ident i f 1 cat  1 on 

- Coordination 

- Implementation 

A next  change/  in  general/  involves  a subset  of  the  system's 
n modules.  Such  a subset  called  R.  can  be  described  by  a 
vector  ■' 


i€l] 


wl,ey,  ViJ^  if  iwvoW 

1^0  if  Hcrf 


j 6 J y 4lte  powe^<“  sei  L 
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R.  described  by  V.  Is  called  ah  i mpact  set.  Once  the  impact 
s4t  induced  by  a Aext  change  is  known/  all  other  modules 
drop  out  of  consideration.  During  the  i dent i f i eat ! on  phase 
effort  is  spent  on  selecting  the  elements  of  the  impact  set 
from  the  candidate  set  i.e.  the  power  set.  The  complexity 
of  this  effort  is  then  simply  the  a priori  uncertainty  or 
the  entropy,  which  Is  removed  by  the  Identification  of  the 
impact  set: 


w Cv)= 


where  P(Vj)  Is  the  probability  of  occurrence  of  Rj . 
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H,  the  usual  formalization  of  uncertaint//  can  be 
interpreted,  for  example,  as  the  minimum  average  number  of 
binary  decisions  necessary  to  find  the  impact  set  R. . This 
measure  suits  well  our  intuition  and  appears  proportional  to 
the  effort  needed  for  identification.  If  R.  is  known  a 
priori,  then  P(V.)*1  and  all  other  impact  sits  have  zero 
probability.  Consequently  H(V)*0  and  there  is  no 
identification  necessary. 

The  maximum  effort  coincides  with  maximum  entropy.  This 
happens  when  ail  impact  sets  are  equally  likely  and 


Furthermore,  the  uncertainty  decreases  with  the  number  of 
modules  or  with  any  departure  of  the  probabilities  from  the 
uniform  distribution.  After  all,  identification  easier 
with  a non-uniform  distribution. 

Once  the  Impact  set  modules  are  collected,  changes  within 
this  set  must  be  coord i nate . Effort  C of  this  phase  is  a 
bit  more  difficult  to  model  although  it  seems  obvious  that 
here  too  the  measure  should  Increase  monoton i cal  1 y with  the 
size  n.of  impact  set  R.  . Hov/ever,  more  research  is  needed  in 
this  afea  and  we  can  o'^fer  here  only  a few  ideas  for 


A.  n^- 


based  on  the  assumption  that  changes  in  modules  must 
be  coordinated  pairwise 


yt; 

-A/  /yyi  ^ 


al ternat i ves : 


where  m is  the  number  of  choices  to  be  made  for  each 
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.noclu  1 a 


name’y  the  number  of  possible  design  sequences  through 
the  set  of  modul es 


Perhaps  the  easiest  to  capture  Is  the  i mo  1 erne 


Phase. 


The  Impact  set  is  already  identified,  ie  change  designed, 
i-.e.  its  individual  components  coordinated.  All  that 
remains  is  to  execute  and  document  it.  (Complications  due  to 
several  change  teams  working  concurrently  are  not  considered 
in  this  paper . ) 

The  most  plausible  way  to  characterize  this  simplest  of  the 
three  phases  is  to  assume  that  work  W spent  is  proprotional 
to  the  impact  set  size 


V 


So  far  our  analysis  of  the  three  phases  of  software 
moc I f i ca t i on  activity  starts  with  the  next  change  assumed  to 
be  given,  while  the  Identity  of  the  impact  set  is  considered 
uncertain.  Now  we  introduce  the  notation  of  the  h i t set ; 


which  is  a collection  of  modules  as  explicitly  presented  at 
next  change  time:  the  resulting  impact  set  must  then  be 
Identified  as  described  earlier. 

The  hit  set  Q|^  can  also  be  described  by  the  vector 
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P(U.)  is  the  probability  of  the  collection  Q.  hitting  the 
system.  ^ 

The  total  modification  uncertainty  over  the  product  set  (or 
matrix)  UxV  can  be  derived  from  their  joint  distribution. 
Using  conditional  probabilities 


thus  decomposing  the  entropy  into  two  terms.  The  first  one 
represents  the  uncertainty  of  the  hit  - characterizing  the 
modification  process;  while  the  second  term  stands  for  the 
additional  uncertainty  about  the  actually  impacted  modules- 
a pure  system  property. 

We  would  also  like  to  emphasize  the  fundamental  difference 
between  Identification  and  coordination  efforts.  They 
contribute  in  quite  different  ways  to  the  difficulties  of 
software  system  modification.  Maximum  coordination,  for 
example,  is  needed  with  the  largest  impact  set  which  may  be 
coupled  with  minimum  identification  entropy:  only  one  Impact 
set  Is  of  size  n.  On  the  other  hand  minimum  (zero) 
coordination  occurs  in  impact  sets  of  unit  size  and  only 
impacts  sets  greater  than  one  have  probability  zero,  which 
still  leaves  the  Identification  effort  to  vary  up  to 
log2(n  + l.) . 

CONCLUDING  COMMENTS 

A good  application  of  the  ideas  presented  In  this  paper 
could  be  to  systematically  test  them  against  the  results  of 
behavioral  studies  <WEI>.  We  also  believe  in  the  relevance 
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of  sor.e  basic  concepts  of  Information  theory  to  software 
complexity.  For  instance,  information  can  be  defined  only 
in  tne  context  of  uncertainty  (namely,  for  an  answer  you 
must  first  have  a question).  Therefore  the  top  down  method 
of  design  may  be  an  oversimplification:  iterative  steps 
between  exploring  the  problem  (generating  uncertainty)  and 
selecting  a particular  solution  to  it  (reducing  uncertainty) 
appear  to  be  closer  to  real  life.  But  more  concrete  and 
immediately  applicable  guidelines  can  be  drawn  from  our 
approach : 


impact  sets  - 
a priori 


try  to  1 oca  1 i ze 


not  completely  known, 
relative  frequency  of 


Des i gn  for  sma  1 1 
potent i al , though 
modi f i cat i ons 
Modularize  systems  by  the 
expected  modifications 
Organize  documentation  by  providing  the  fastest 
access  to  areas  which  most  likely  change 
Monitor  hit  sets  and  impact  sets  during  maintenance 
for  planning 


i, 

1. 
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Potential  Impacts  of  Software  Science  on 
Software  Life  Cycle  Management 


M.  H.  Halstead 
Purdue  University 


Abstract 

A listing  of  present  needs  in  Software  Engineering  is  followed  by 
a brief  discussion  of  new  and  "quasi -complete"  engineering  disciplines 
and  their  relation  to  corresponding  branches  of  natural  science. 

Recent  results  in  Software  Science  are  then  described,  suggesting  a 
natural  science  base.  The  paper  concludes  with  specific  suggestions 
of  areas  in  which  these  results  might  well  provide  guidance  and  insight 
into  problems  of  software  development  and  maintenance. 
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Engineering  Needs 


The  successful  management  of  large  prograimlng  projects  over  their 
complete  life  cycles  depends  largely  upon  the  discipline  of  Software 
Engineering.  But  In  Its  present  state  of  development  this  discipline 
still  falls  far  short  of  being  a "quasi -complete"  branch  of  engineering. 

Substantial  progress  Is  urgently  needed  on  many  fronts.  A handful 
might  be  mentioned.  1).  Problm  Spaaifioaticna.  Software  Engineering 
should  be  able  to  provide  optimal  methods  for  obtaining  them,  resolving 
their  ambiguities  and  Incompatibilities,  determining  their  completeness, 
and  the  "best"  way  to  present  them  to  prograimters.  2).  Programner 
Productivity.  In  this  area,  a "quasi -compl ete"  discipline  would  provide 
guidelines  for  programmer  selection,  training,  and  subsequer^t  evaluation; 
as  well  as  quantitative  methods  for  estimating  the  man-hours  and  organi- 
zation required  to  achieve  any  well  specified  goal.  3).  Progvcm  Teating. 
Quantitative  methods  are  needed  for  estimating  the  effects  of  source 
language,  modularity,  program  level  or  complexity,  volume  and  program 
clarity  upon  program  testing  and  maintenance.  4).  Job  Scheduling.  More 
reliable  techniques  are  needed  for  estimating  man-power  requirements  of 

a project  as  a function  of  specifications,  progrannlng  language,  team  size 
and  nrlx,  memory  constraints  and  required  reliability.  5).  Optimiaing 

Jmplmentation  veraua  Mceintemnce  eoata.  A quasi -compl  ete  engineering 

discipline  should  provide  quantitative  guidelines  for  sound  tradeoff 

studies  In  this  Important  but  complex  area. 

Natural  Science  and  Engineering 

For  any  engineering  discipline  to  be  quasl-complete.  It  must  rest  upon 
and  be  grounded  In  a "hard"  natural  science,  a science  with  sound  metrics. 


reproducible  experiments,  and  dependable  "laws".  In  virtually  all  cases, 
branches  of  engineering  have  preceded  (and  perhaps  stimulated)  the 
natural  science  upon  which  they  are  now  based.  During  that  time,  however, 
their  value  to  mankind  was  severaly  limited.  But  now,  for  example, 
aeronautical  engineering  based  on  fluid  dynamics,  power  engineering  based 
on  thermodynamics,  electrical  engineering  based  on  electrodynamics,  and 
mechanical  engineering  based  on  statics,  dynamics  and  strength  of 
materials  may  all  be  considered  quasi -complete,  highly  competent,  and 
useful . 

Software  Science 

A considerable  body  of  evidence  now  exists  which  suggests  that  the 
metrics,  methods,  and  hypotheses  of  software  science  [7]  may  be  capable 
of  providing  such  a base  for  software  engineering.  It  Is  pertinent  to 
noter  however,  that  theories  are  not  theorems.  They  require  Independent 
experimental  confirmation  at  more  than  one  laboratory.  Unlike  mere 
mathematical  models,  theories  must  provide  new  Insight  Into  natural 
phenomena,  and  they  only  become  Important  when  they  are  shown  to  predict 
previously  unrecognized  and  unexpected  relationships  in  areas  beyond  their 
originally  Intended  scope.  Further,  a theory  Is  never  complete,  but 
continues  to  be  used  only  until  Its  recognized  Inadequacies  can  be 
eliminated  by  a new  theory. 

Software  science  Is  based  on  a handful  of  language  Independent  para- 
meters which  can  be  measured  (or  counted)  directly  from  any  hard  copy  or 
computer  progrmn.  These  are  the  number  of  unique  operators  (n^),  the 
number  of  unique  operands  the  total  usage  of  operators  (N^);  and 


388 


the  total  usage  of  operands  (N^).  A fifth  parameter,  the  number  of 
conceptually  unique  Input/output  operands  (n^*)  required  by  a procedure 
call  upon  the  program  is  also  an  important  language-independent  metric. 

These  basic  metrics  are  not  independent,  and  a number  of  quite 
useful  relationships  have  been  found  among  them.  For  example,  denoting 
the  sum  of  and  as  the  length  N,  it  has  been  found  that  programs 
tend  to  obey  the  relation 

N ■ log^  log2  02 

Further,  denoting  the  sum  of  and  as  the  vocabulary  n.  the 
program  volume  V is 

V » N log^  n 

The  potential  (or  least  possible)  voline  V*  is 

V*  » (2  -r  02*)  log2  (2  02*) 

which  gives  an  implementation  level  L of 

L = V*/V  « ^ 3.2- 

O2 

It  follows  that  the  product 
V*  * LY 

is  invariant  under  translation  from  one  language  to  another. 

It  has  also  been  found  that  the  number  of  elementary  mental  dis- 
criminations (E)  required  to  produce  a program  should  be  given  by 

E « V/L 

This  leads  directly  to  an  estimate  of  programming  time  (T).  Using 
the  Stroud  Number  (S)  or  18  elementary  mental  discriminations  per 
second,  gives 

T « E/S  - V/SL 
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Initially  unforseen  relationships  derivable  from  the  basic 
metrics  Include  Ostapko's  [10]  derivation  of  Rent's  Rule  for  circuit 
to  pin  ratios  in  hardware.  Elci's  [3]  demonstration  that  the  lengths  of 
operating  systems  are  functionally  related  to  the  number  of  their 
allocatable  resources,  and  Funami’s  [5]  demonstration  that  progranming 
error  rates  are  related  to  E.  Perhaps  the  most  interesting  unexpected 
finding  to  date  is  the  observation  that  the  relationships  governing 
computer  programs  can  be  applied  to  technical  prose  as  well. 

With  respect  to  deeper  understanding  or  insight,  one  might  list 
the  areas  of  program  purity  or  "impurity  classes",  the  role  of  modularity, 
the  quantitative  effects  of  "GO-TO's",  the  measurement  of  clarity,  and 
most  recently  some  apparent  insight  into  the  learning  process  itself. 

Independent  experimental  verifications  of  various  facets  of  the  i 

overall  theory  have  been  published  by  Bohrer  [2]  of  Illinois,  Elshof  [4] 
of  General  Motors,  Bell  and  Sullivan  [1]  of  Mitre,  Ostapko  [10]  of  IBM 
and  Love  and  Bowman  [9]  of  General  Electric. 

Experimental  Methodology 

To  illustrate  the  relationships  and  methodology  discussed  above,  we 
will  first  present  the  results  and  analysis  of  a simple  experiment,  and 
follow  with  a few  summaries  of  previously  published  data. 

In  January  1977  a class  of  28  advanced  graduate  students  at  Purdue 
individually  programned  Euclid's  greatest  common  divisor  algorithm  in 
Fortran,  and  counted  the  software  parameters  in  their  own  versions.  Their 
results  are  given  in  Table  1.  Students  10,  16  and  27  neglected  the 
implied  "End-of-line"  operator  in  Fortran,  so  their  reported  values  of  '5 

have  been  increased  by  one,  and  their  values  of  have  been  increased  by  12. 

I 

i J 


n 


i 
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Table  1 . Software  Parameters  of  28 

Independent  Fortran  Versions 
of  the  6CD  Algorithm 


Student 

Ni 

^2 

Student 

Nz 

1 

11 

6 

34 

21 

15 

11 

6 

34 

21 

2 

11 

5 

32 

19 

16 

12 

7 

34 

21 

3 

12 

6 

34 

21 

17 

12 

6 

38 

21 

4 

12 

6 

34 

21 

18 

11 

6 

33 

21 

5 

10 

6 

31 

21 

19 

12 

6 

34 

21 

6 

12 

6 

34 

21 

20 

11 

6 

34 

21 

7 

11 

6 

34 

21 

21 

10 

6 

34 

21 

8 

11 

6 

31 

21 

22 

11 

6 

34 

21 

9 

12 

6 

34 

21 

23 

11 

6 

32 

21 

10 

12 

6 

36 

21 

24 

12 

6 

35 

21 

11 

11 

6 

35 

21 

25 

12 

5 

33 

19 

12 

11 

6 

34 

21 

26 

12 

6 

32 

21 

13 

11 

5 

33 

19 

27 

11 

6 

34 

21 

14 

12 

6 

34 

21 

28 

10 

6 

31 

21 

MEANS 

11.32 

5.93 

33.64 

20.79 

S.O. 

.67 

.38 

1.50 

.63 

Using  the  Individual  values  In  Table  1.  a number  of  the  software 
relationships  can  be  evaluated*  and  the  degree  of  conformity  calculated. 
Length 

Obtaining  the  observed  value  of  length  (N)  from 
N • + Nj 

and  the  estimated  length  (A)  from 

A 

N ■ log^ 
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gives  mean  values  of 

N » 54.43  + 1.75 
N » 54.90  + 3.76 
(N-N)/N  • -0.009  + 0.058 
Implementation  Level 

Obtaining  the  observed  potential  volume  (V*)  from  the  condition 
that  a procedure  call  on  a GCO  algorithm  must  have  two  inputs  and  one 
output,  or  n*  • 3 in 

V*  * (2  + n^*)  log^  (2  + n^*) 

and  the  observed  volume  (V)  from 

V - (Nj  + N^)  log^  (nj  + n^) 
allows  the  observed  level  (L)  to  be  calculated  from 

L » V*/V 

The  estimated  level  (L)  is  obtained  from 


L « i-  ^2- 

Then  for  Table  1,  the  mean  values  are 


L » 0.0529  + 0.0023 
L - 0.0505  + 0.0035 
(L-L)/L  • 0.028  1 0.067 


Potential  Volume 

A 

Obtaining  the  estimated  potential  volume  (V*)  from 

and  the  actual  potential  volume  (V*)  from  ■ 3 yields 

V*  ■ 11.61  + 0.00 
V*  - 11.28  + 0.78 
(V*-V*)/V*  - 0.02**  + 0.067 
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ii 

ii 
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Vocabular 


Obtaining  the  observed  vocabulary  (n)  from 

n » 

and  the  estimated  vocabulary  (n)  by  solving  iteratively  for  n as  a 


function  of  N in 


gives  the  mean  values 


• n log^  (V2) 


n » 17.25  + 0.80 
n(N)  * 17.42  + 0.38 
(n-n(N))/n  » -0.016  + 0.038 


Unique  Operators 


Using  the  vocabulary  n(N}  estimated  from  length  as  calculated 

A 

above,  the  estimated  unique  operator  count  t1j(N)  can  be  obtained  from 

» (n-B)/(A+l) 


where 


Hz*  loq2(n^*/2) 
2 n-* 


; B » - 2A 


The  data  in  Table  1 give  the  following  average  values 

• 11.31  +0.67 

nj(N)  - 11.20  + 0.28 

(ni-nj(N))/nj  ■ 0.008  + 0.055 
Unique  Operands 

In  the  same  way,  the  observed  n2  can  be  estimated  from  ^2* 
and  length  via 

n2(N)  - (An(N)  + B)/(A  + 1) 
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And  again,  the  Table  1 data  show 

” 5.93  + 0.38 
n2(N)  - 6.23  + 0.10 

“ -0.054  + 0.064 

Surrmarizing  the  relative  errors,  we  have 


Length  (N) 

-0.009  + 0.058 

Level  (L) 

0.028  + 0.067 

Potential  Volume  (V*) 

0.028  +0.067 

Vocabulary  (n) 

-0.016  + 0.038 

Operators  (hj) 

0.008  + 0.055 

Operands  (oj) 

-0.054  + 0.064 

This  can  be  taken  as  evidence  that  for  a very  small  program, 
replicated  by  28  highly  fluent  programmers,  the  software  hypotheses 
tested  gave  reasonably  good  agreement  with  the  observations. 

In  order  to  illustrate  the  applicability  of  these  relationships  to 
programs  large  enough  to  be  of  practical  interest,  Elshof's  [4]  data 
validating  the  length  equation  for  conmercial  PL/1  programs  are  given 
in  Table  2. 

Table  2.  Elshof's  PL/1  Data 


Number  of 
Programs  in 

Size  Class 

Length 

Observed  (N) 

Length 

Predicted  (N) 

3 

18.592 

19,091 

17 

10,685 

11,049 

23 

5,751 

6,005 

39 

3,165 

3,318 

17 

1,590 

1,663 

n 

831 

911 

• • 

4 

369 

522 

5 

198 

195 

1 

122 

129 

Totals  120 

41 ,303 

42,883 

f ' 

« • 

394 


( 

; 


pia-UR.€  /.  FLEMCMTARt  MCMTAk  Diaci^l  MINATloNi 
O G-o^ooAa-H«u«rcAO  Data 

VC  Johajsoai  Data 
n w/ik5TOA/- Fck/ X Data 


Figure  1 has  been  taken  front  another  paper  [8],  which  indicates 
that  projects  ranging  from  well  under  one  day  to  well  over  one  hundred 
man  years  follow  the  Effort  Equation  in  a general  way. 

Indicated  Studies 

A number  of  areas  in  which  software  science  might  prove  useful  to 
Software  Life  Cycle  Management  come  immediately  to  mind.  While  each  of 
the  five  areas  cited  in  the  introduction  as  needing  substantial  improve- 
ment should  benefit  directly,  we  can  perhaps  be  more  specific. 

With  respect  to  task  speeifiaations^  for  example,  one  might  suggest 
five  steps. 

1)  Start  with  small  tasks,  requiring  only  one  programmer  from  10  minutes 
to  8 hours  to  implement,  and  gather  a substantial  number  of  samples. 

2)  Analyze  the  technical  English  problem  statements,  obtaining  n,  N,  V, 

L and  Y*,  E and  T. 

3)  Perform  a similar  analysis  of  the  resulting  programs. 

4)  Study  the  effect  of  different  methods  and  techniques  of  problem 
statement  on  the  resultant  small  programs. 

5)  Expand  the  study  to  large  programs. 

Similarly,  programmer  productivity  relationships  are  sufficiently  important 
to  warrant  large  scale  investigations. 

1)  Repeat  experiments  to  determine  individual  programmer  variances 
between  actual  and  calculated  programning  times. 

A 

2)  Note  that  the  calculated  values  of  T for  very  large  programs  have  all 
been  based  upon  average  values  of  language  level.  Because  deviations 
from  average  would  be  expected  to  contribute  to  the  observed  variance 
in  T,  it  should  be  illuminating  to  actually  measure  this  effect. 


396 


3)  For  a significantly  large  data  base,  obtain  the  statistical  variance 


between  observed  and  calculated  prograiming  times. 

4)  Perform  quantitative  studies  on  the  effect  of  E of  existing  programs 
on  the  rate  of  error  discovery  by  new  programmers.  This  should  yield 
a measure  of  their  fluency  (and  concentration). 

5)  Investigate  the  possibility  that  programming  aptitude  might  be 
estimated  by  a software  analysis  of  a technical  prose  paragraph 
written  by  a candidate  for  programmer  training. 

The  area  of  program  testing  might  benefit  by  further  investigations 
of  the  software  relationships.  This  might  require 
1)  • Sharpening  the  definition  of  "Delivered"  bugs. 

2}  Development  of  a large  data  base,  with  samples  from  most  of  the 
widely  used  languages. 

3)  Repetition  of  experiments  to  determine  the  expected  variance  between 
observed  and  calculated  error  rates. 

4)  Analysis  of  modularity,  following  Zislis'  [11]  use  of  software  science 
for  program  testing. 

5)  Use  of  software  error  rate  relations  in  predicting  remaining  errors 
as  a function  of  expected  errors  and  errors  removed. 

The  area  of  dob  aoheduling  is  related  to  that  of  progranmer 
productivity,  but  requires  other  information  as  well.  Consequently,  it 
involves  a number  of  additional  points. 

1)  Because  any  two  independent  software  parameters  determine  all  others, 
it  follows  that  the  task  specifications  and  the  language  to  be  used 
determine,  in  principle,  the  time  to  be  required.  In  practice. 


however,  it  is  not  that  simple.  For  example,  even  a concentrating 


programmer,  fluent  in  the  language,  without  computer  memory  con- 
straints, must  start  with  a complete  problem  statement.  Furthermore, 
a problem  statement  which  contains  no  contradictions  or  ambiguities 
may  be  "complete"  for  one  programmer,  but  not  for  another.  Never- 
theless, the  existence  of  a basic  relationship  between  the  number 
of  conceptually  unique  Input/output  operands  (n2*)>  the  language 
level  (X),  and  the  time  (T)  suggests  that  an  intensive  investigation 
is  now  possible  and  warranted. 

2)  It  has  been  observed  that  the  product  LV  ■ V*  is  invariant  when  a 
given  algorithm  is  translated  from  one  language  to  another.  But 
within  any  one  language,  L decreases  as  the  potential  volume  in- 
creases. Consequently,  a given  language  can  be  characterized  by  its 
language  level  (x),  defined  as 
X » LV* 

Algebraically,  this  results  in 

T • V/SL  • V*3/SX2 

and  consequently  x is  a parameter  of  considerable  interest.  While 
the  mean  value  of  x appears  to  lie  somewhere  near  one  for  a number 
of  programming  languages,  it  has  a large  variance  which  appears  to 
increase  as  the  mean  increases.  Because  the  data  so  far  available  is 
based  on  small  samples  of  small  programs,  it  can  not  be  used  with 
confidence.  Therefore,  statistical  determinations  of  means  and 
variances  of  X for  any  language  of  Interest  should  be  made.  This 
study  might  well  Include  investigations  of  the  effect  of  different 
progranmlng  methodologies  within  a single  language.  It  could  hi  so 
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be  extended  to  an  analysis  of  any  proposed  new  language.  It  could  I 

then  serve  in  trade-off  studies  on  cost  versus  benefits  of  change 
to  a higher  level  language,  or  the  question  of  special  purpose 
versus  general  purpose  languages.  | 

With  respect  to  the  problems  of  optimizing  inplzmentation  ooata  | 

versus  maintencmae  oosts,  it  appears  quite  likely  that  the  quantitative  | 

approach  provided  by  software  science  can  be  of  considerable  value. 

This  results  from  recent  work  of  Gordon  [6],  who  has  shown  that  the 

measure  of  elementary  discriminations  E is  in  an  interesting  sense 

ambiguous.  : 

In  the  usual  case  of  program  implementation,  E does  indeed  I 

measure  the  time  required  to  develop  the  program.  If,  however,  additional  1 

time  is  then  spent  in  improving  or  increasing  the  legibility  of  the  | 

program,  the  effect  is  to  reduce  the  final  value  of  E,  rather  than  to  | 

increase  it.  The  final  value  of  E for  an  improved  program  then  | 

represents  not  the  total  effort  to  write  it,  but  a measure  of  the  effort  I 

f 

to  understand  it  — a measure  of  clarity. 

This  suggests  that  a quantitative  measure  of  clarity  could  be  made 
before  a program  is  polished.  Then,  the  amount  of  effort  which  could 
advantageously  be  used  in  increasing  the  clarity  could  be  determined  on 
the  basis  of  the  needs  of  maintenance. 


f 


l 
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Notes  on  Software  Reliability  and  Quality 


1.  Introduction 

The  notes  are  Intended  to  review  the  subjects  of  software  reliability 
and  quality  from  a managerial  perspective.  As  such  the  notes  are  In- 
tended to  round  out  the  coverage  of  the  topics  provided  by  experts  at 
the  USACSC  workshop. 

The  managerial  perspective  Is  the  author's:  the  focus  Is  on  projects  of 
substantial  scale  where  formality  of  organization  and  approach  are 
required.  More  than  one  manager  will  be  Involved  and  their  principle 
tasks  are: 


o balancing  product,  resources,  and  schedule 

o establishing  and  maintaining  a process  which  affords  the 
visibility  to  appraise  and  redirect  progress. 

The  second  principle  task  orients  the  author's  research  In  software  pro- 
cess engineering. 


The  notes  are  organized  Into  sections  which  comment  on  software  relia- 
bility, system  perspective,  quality  assurance,  a quality  Index,  and 
data.  The  software  reliability  note  Is  adapted  from  a 1975  note  by  a 
managerial  colleague,  Donald  O'Neill.  If  the  software  reliability  note 
appears  Inconclusive,  then  It  has  succeeded,  since  no  approach  to  software 
reliability  appears  to  be  preeminent.  The  brief  system  perspective  note 
suggests  an  alternative  approach  to  software  reliability:  addressing  the 
larger  question  of  system  reliability.  The  quality  assurance  note 
criticizes  the  formal  quality  assurance  programs  required  for  U.S.  weapon 
system  development.  The  quality  Index  note  examines  a proposed  Index. 
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The  final  note  Issues  an  appeal  for  collecting  and  publishing  data  to 
support  software  reliability  and  quality  research. 

2.  Software  Reliability 

Reliable  software  Is  software  that  does  not  fall.  However,  large  and 
complex  software  systems  sometimes  fall.  The  question  Is  not  whether 
the  software  Is  reliable,  but  rather  how  often  It  can  be  expected  to 
fall. 

The  temptation  to  define  software  reliability  In  the  well  understood 
framework  of  hardware  reliability  must  be  resisted  until  the  salient 
differences  between  hardware  and  software  are  examined.  All  soft* 
ware  errors  contribute  to  software  reliability.  Given  the  same  data  and 
conditions  software  operations  (errors)  are  repeatable.  The  difficulty 
In  recreating  the  same  conditions  (data  and  timing)  Is  such  that  a pro- 
blem log  must  retain  all  unresolved  problems  so  that  multiple  occurrences 
of  an  Infrequently  appearing  software  error  can  be  correlated  to  diagnose 
and  correct  the  defect. 

At  Initial  operation,  hardware  reliability  may  exceed  software  reliability. 
However,  as  software  faults  are  corrected,  the  software  reliability  should 
exceed  hardware  reliability. 

Certain  boundaries  must  be  placed  on  the  conditions  under  which  software 
reliability  Is  appraised.  In  order  for  a system  error  to  be  charged  to 
software.  It  must  occur  when  all  hardware  Is  operating  correctly.  For 
example,  the  execution  of  an  Illegal  Instruction  due  to  a hardware  tran- 
sient resulting  In  a subsequent  propagation  of  errors  through  software 
should  not  be  attributable  to  software.  To  some  degree,  software  can 
compensate  for  transient  hardware  errors  by  defensive  coding,  (e.g.,  re- 
trying a failed  element) . In  addition,  software  reliability  measures 
should  be  limited  to  testing  or  using  software  within  the  limits  and 
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constraints  of  a design  envelope.  If  these  limits  are  exceeded,  the  soft- 
ware should  not  be  charged  for  performance  related  errors  resulting  from 
the  out-of -specification  conditions. 

In  developing  software  many  Instructions  are  written.  There  Is  potential 
for  Introducing  error  In  each  Instruction,  (e.g.,  compares  and  branch 
Instructions  used  In  Implementing  flow  of  control  are  most  error  prone; 
followed  by  Instructions  used  for  data  operations) . When  the  code  Is  exe- 
cuted, some  segments  are  executed  often,  others  Infrequently.  Through 
use  the  software  most  executed  becomes  most  reliable.  It  takes  longer 
to  uncover  problems  In  seldom  used  segments. 

Software  reliability  can  be  approached  from  two  directions:  past  success 
and  future  failure.  By  evaluating  past  success,  the  proportion  of  the 
system  that  has  been  successfully  demonstrated  Is  determined.  Software, 
unlike  hardware,  can  be  depended  upon  all  the  time  to  perform  correctly 
In  operational  use  for  those  areas  demonstrated.  The  test  program  pro- 
vides the  means  for  establishing  proven  capability.  Since  software  Is 
repeatable,  a test  program  that  successfully  exercises  the  operational 
scenario  certifies  the  software  reliability.  Although  every  system 
Instruction  can  be  tested.  It  Is  not  possible  to  test  every  combination 
of  data  and  timing  variations.  However,  the  concept  that  reliability  can 
be  calibrated  through  the  relationship  of  test  to  operation  has  real 
value. 

A test  coverage  matrix  can  help  evaluate  effectiveness.  The  test  program 
requires,  In  effect,  a test  of  the  test  program.  A technique  for  testing 
the  test  procedures  Is  comprehensive  code  reading  following  each  test 
phase.  The  effectiveness  of  the  test  program  can  be  judged  by  the  number 
of  additional  errors  detected. 
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The  approach  to  quantifying  failures  uses  past  failures  to  project  future 
failures.  The  key  is  to  record  and  categorize  the  errors  detected  in 
each  phase  of  development.  Since  the  number  of  errors  encountered  is  a 
function  of  the  number  of  errors  still  in  the  software,  then  this  record 
predicts  future  failures. 

A simple  error  removal  model  is  suggested:  some  fraction  of  the  remaining 
error  population  is  identified  and  corrected  by  each  distinct  software 
testing  activity.  Each  distinct  activity  must  add  a new  dimension 
rather  than  provide  more-of-the-same  testing.  This  simple  model  uses 
two  factors:  the  error  removal  rate  and  the  number  of  testing  steps. 

The  error  removal  model  confirms  that  top-down  approaches  should  increase 
system  reliability.  If  there  are  N test  periods,  the  first  Increment 
(system  control)  is  tested  N times,  the  second  increment  is  tested  N-1 
times,  with  the  last  Increment  being  tested  perhaps  only  once. 

For  any  system  there  is  a reliability  figure  Independent  of  whether  it 
is  known.  The  notion  of  dependability  requires  that  the  software  reli- 
ability figure  be  known,  and  the  real  significance  of  determining  or 
predicting  the  software  reliability  lies  in  the  use  of  error  information. 
If  decisions  need  to  be  made  based  on  the  level  of  software  quality 
assurance,  then  consideration  should  be  given  to  establishing  permissible 
software  error  budgets  that  must  be  achieved.  It  may,  in  fact,  be  more 
effective  to  manage  the  error  budget  than  attempt  to  predict  the  ultimate 
reliability.  The  penalty  for  low  reliability  is  operational  Impact  and 
high  maintenance  cost.  However,  the  cost  and  schedule  impact  resulting 
from  excessively  high  software  reliability  emphasis  may  be  unnecessary. 

By  establishing  the  required  objective  and  managing  to  it,  the  resulting 
software  satisfies  the  operational  need  within  the  cost  and  schedule 
boundaries. 
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Another  consideration  Is  the  effect  of  the  software  error  on  the  opera- 
tion of  the  system.  A catastrophic  error  (due  to  a computer  stop,  soft- 
ware Interlock,  memory  destruction,  or  Input/output  difficulty)  results 
In  the  Inability  of  software  to  continue  processing.  A second  level  of 
software  error  Is  characterized  by  the  continuation  of  the  processing 
following  the  error  but  with  an  unknown  effect  on  the  results.  The 
effect  of  a software  error  may  be  predictable  and  minimal.  An  error 
severity  category  may  be  useful  In  terms  of  determining  where  the  test 
emphasis  should  be.  A comprehensive  test  program  should  remove  all 
catastrophic  errors. 

Error  prone  segments  account  for  a large  proportion  of  the  life  cycle 
errors.  For  example,  Jones reports  about  4X  of  the  modules  account 
for  perhaps  38X  of  all  OS/360  defects.  Traditionally,  error  prone  seg- 
ments have  had  complex  flow  of  control  (simplified  now  through  structured 
programming)  or  complex  data  Interfaces  to  other  segments.  Identifying 
error  prone  segments  early  In  the  process  substantially  Improves  relia- 
bility through  remedial  action  and  test  emphasis. 

3.  System  Perspective 

We  might  attack  software  reliability  by  determining  measures  for  system 
reliability.  At  present,  system  reliability  calculations  take  only  hard- 
ware Into  account.  The  Inherent  assumption  Is  perfect  software  relia- 
bility. As  a result  there  Is  no  margin  for  software  error  since  the 
reliability  budget  usually  has  been  consumed  with  expected  hardware 
failures.  While  the  hardware  errors  may  be  transient.  Intermittent,  or 
solid,  only  solid  errors  form  the  basis  for  the  reliability  calculation. 

A simple  example  Illustrates  the  possibility  of  a system  perspective. 

9 

Devices  satisfying  a specification  of  one  bit  lost  In  10  bit  transfers 
were  attached  to  a computer  which  demanded  10^  bits  per  second  without 
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parity  checking.  System  failures  which  occurred  approximately  every  15 
minutes  were  attributed  to  software.  A recovery  procedure  Implemented 
in  software  eliminated  system  failures  from  this  source.  We  should  re- 
sist the  urge  to  place  all  the  blame  on  the  designers  because  the  com- 
ponents met  their  individual  solid  failure  budgets  and  the  designers  had 
little  help  from  traditional  hardware  reliability  models.  The  software 
as  originally  Implemented  also  met  its  original  specification.  Further- 
more, while  the  example  has  been  presented  to  highlight  the  problem 
source,  the  "intermittent"  problem  was  difficult  to  diagnose  from  the 
evidence  in  memory  dumps  which  were  not  always  taken  because  of  the 
pressures  attendent  to  system  integration. 

Another  example  may  help  illuminate  the  system  perspective.  In  the 
design  of  a high  availability  system,  two  computers  were  specified:  one 
active,  the  other  a "hot"  standby.  Following  a switchover,  the  design 
called  for  the  "failed"  computer  to  be  tried,  without  repair,  as  the 
standby.  The  rationale  was  simply  intuition  concerning  the  relative 
frequency  of  non-solid  and  solid  ha.dware  faults.  The  intuition  appears 
justified  in  operation  where  recoveries,  including  those  required  by  the 
standby  system,  exceed  repairs. 

The  system  perspective  suggests  several  questions  concerning  hardware, 
software,  and  system  reliability  measures.  Some  of  these  questions 
warrant  discussion  during  the  workshop: 

o how  much  emphasis  should  system  reliability  be  given? 

o can  system  reliability  considerations  guide  software  relia- 
bility research? 

o should  we  perceive  software  as  the  system  and  somehow  sub- 
ordinate hardware? 


4.  Quality  Assurance^  ' 

One  quality  assurance  (QA)  definition  is  the  formal  approach  to  syste- 
matically check  the  development  of  a product  to  assure  the  product  con- 
forms to  requirements.  In  this  definitional  context,  quality  assurance 
can  be  paraphrased  as  conformity  assurance  or  adequacy  assurance. 

(3) 

One  military  specification  identifies  several  software  QA  program 
areas: 


o work  tasking  and  authorization  procedures 
o Configuration  management  and  library  control 
o software  documentation 
o reviews  and  audits 
o testing 

o corrective  action 
o tools,  techniques,  methodologies 

Distinction  between  QA  and  testing  is  attempted  in  some  contractxial 
terms.  The  distinction  is  usually  organizational  with  QA  given  an  auto- 
nomous posture  which  frees  QA  from  the  demands  of  delivery  schedules. 

The  distinction  is  difficult  below  the  grossest  level  of  detail.  The 
figure  attempts  to  distinguish  quality  reviews  in  a project  context. 

The  figure  identifies  two  types  of  quality  reviews:  one  associated 
with  design  reviews  and  the  other  associated  with  testing. 

Since  design  reviews  are  a substitute  for  objective  execution  tests,  the 
associated  quality  review  requires  specific  focus  to  avoid  duplicative 
effort.  The  QA  focus  must  establish,  first,  a role  as  requirements 
advocate  to  assure  the  requirements  are  not  lost  in  assessing  technical 
and  Implementatlonal  feasibility  of  the  system  design;  and,  second,  a 
role  as  test  reviewer  to  assure  the  adequacy  of  the  test  design. 
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The  quality  review  associated  with  testing  Is  more  mechanical  chan  the 
design  associated  review,  because  QA  Is  more  nearly  a witness  than  a 
participant  role.  The  quality  review  assures  the  test  environment 
(hardware,  software  procedures,  data)  and  test  results. 

One  of  the  central  QA  problems  Is  the  definition  of  software  quality. 

Two  schools  of  thought  concerning  software  quality  are: 

o Quality  Is  measured  by  Che  number  of  software  errors  reported 
by  users. 

o Quality  Is  measured  by  the  length  of  time  software  runs 
without  falling  (stopping) . 

The  first  has  the  difficulty  of  either  having  no  error  budget  (Implicitly 
zero  defects)  and  having  only  negative  evaluation  as  errors  are  detected; 
or  having  an  error  budget  and  having  to  admit  frailty  In  advance.  The 
second  has  the  difficulties  associated  with  software  reliability  discussed 
previously.  In  spite  of  these  difficulties  some  guidance  Is  derived; 
the  QA  activity  must  Include: 

o Procedures  for  reporting  detected  errors 
o Closed  loop  means  to  ensure  error  correction 

An  often  overlooked  QA  problem  Is  accommodating  customer,  contractor,  or 
computer  manufacturer  furnished  software. 

In  addition  to  the  software  Itself,  QA  cognizance  Includes  documentation. 
In  some  projects,  QA  cognizance  Is  limited  to  documentation  and  In  de- 
generate cases  Is  an  editing  function. 

The  software  development  process  Itself  must  Implement  the  QA  discipline. 
This  maxim  should  be  discussed  during  the  workshop  to  determine  Its  value 
In  appraising  or  disciplining  research. 
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The  Implementatlonal  difficulties  also  arise  in  distinguishing  user 
perceived  functions  versus  components  used  in  implementing  function  and 
in  computing  availability.  By  assuming  catastrophic  errors  have  been 
removed  prior  to  operational  use,  a gross  approximation  of  A may  be  made 
based  on  system  availability.  Under  this  assumption  an  essentially 
available  system  quality  would  be  influenced  only  by  corrective  maintenance: 


No  difficulty  is  forseen  in  establishing  cost  accounts  to  distinguish 
corrective  activity  from  enhancement  or  other  supportive  activities. 

Substituting  for  A and  M (equations  1-3)  we  obtain 


(4) 


This  form  shows  flaws  in  the  construction.  For  a software  system  of  a 
given  availability,  as  the  malntainer,  we  can  maximize  Q by  doing  nothing, 
i.e.,  C-0  or  by  having  a total  support  budget  B » C,  e.g.,  get  as  much 
enhancement  budget  as  possible  and  keep  C marginal.  Further,  in  the 
interest  of  maximizing  Q,  we  would  not  fix  a problem  to  restore  f if 

cost  to  fix  f 
c ^ i 


since  to  do  so  would  increase  denominator  more  than  numerator  in  (4) . 

This  could  be  avoided  if  the  "cost  to  the  user"  of  not  having  f^  repaired 
were  reflected  in  some  recurring  cummulatlve  fashion  - perhaps  f^t^  where 
t^  is  time  left  unrepaired. 
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5.  A Quality  Index 


This  note  explores  the  potential  use  of  functional  availability  and 
corrective  maintenance  cost  as  quality  measures.  The  intent  is  to  sti- 
mulate rather  than  to  recommend  the  specific  quality  measure  which  is 
presented. 

S . 1 Operational  Quality 


Cave^  has  suggested  a measure  of  operational  software  quality  based  on 
functional  availability  A and  maintenance  cost  M: 


(1)  Q 


A 

1 + M 


where  (2)  A >■  E 


‘i 

1 - Z f 


Importance  of  1th  function 


availability  of  ith  function 


and  (3) 


M - £ 

M - - 


C • cost  of  corrective  maintenance 
B * total  support  budget 

The  suggested  measure  is  appealing  because  of  its  conceptual  simplicity 
and  the  general  interest  in  the  measures  of  A and  H.  Availability  is  of 
critical  interest  to  the  user  and  to  the  developer  and  its  component 
functional  importance  is  needed  by  the  designer  of  the  various  reduced 
function  modes  in  which  a complex  system  may  operate.  The  ratio  of 
corrective  maintenance  to  total  support  budget  is  of  interest  to  both 
managers  and  researchers. 
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The  suggested  measure  has  difficulties  in  compariative  use  and  in  imple- 
mentation. Two  systems  which  have  identical  quality  indices  can  not  be 
compared.  For  example: 


^1 

^2 


“ 0.45,  however 

0.45 
1 + 0 

0.9 
1 + 1 


The  construction  suffers  from  the  implied  requirement  of  a prior 

definition/identification  of  all  "functions"  and  their  importance. 
Alternatively,  the  mere  existence  of  an  open  (unduplicated)  problem 
report  E implies  a 'function'  denied  (a^  > 0}  and  the  priority  assigned 
to  E is  a measure  of  the  Importance  of  the  function  (f^)  to  the  user. 
Latent  problems,  it  can  be  argued,  are  quality  measures  on  the  software 
producer  but  not  on  the  system  from  the  user's  viewpoint,  l.e.,  the 
availability  of  an  unused  function  is  immaterial.  Therefore  we  can 
indicate  quality  in  some  form  like 

Q - f(E) 

and 

1 . 100 

liTT 

or  more  generally 

A • •= — where  p.  is  a normalized  E priority 
ip^E^  +1  i 

The  disadvantage  of  this  particular  representation  is  that  A decreases 
too  rapidly  with  A better  scaling  could  be  provided  by  klp^E^, 
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The  Implementatlonal  difficulties  also  arise  in  distinguishing  user 
perceived  functions  versus  components  used  in  implementing  function  and 
in  computing  availability.  By  assuming  catastrophic  errors  have  been 
removed  prior  to  operational  use,  a gross  approximation  of  A may  be  made 
based  on  system  availability.  Under  this  assumption  an  essentially 
available  system  quality  would  be  Influenced  only  by  corrective  maintenance: 


1 + M 


No  difficulty  is  forseen  in  establishing  cost  accounts  to  distinguish 
corrective  activity  from  enhancement  or  other  supportive  activities. 


Substituting  for  A and  M (equations  1^3)  we  obtain 
(4)  Q 


-I 


This  form  shows  flaws  in  the  construction.  For  a software  system  of  a 
given  availability,  as  the  maintainer,  we  can  maximize  Q by  doing  nothing, 
i.e.,  C-0  or  by  having  a total  support  budget  B » C,  e.g.,  get  as  much 
enhancement  budget  as  possible  and  keep  C marginal.  Further,  in  the 


Interest  of  maximizing  Q,  we  would  not  fix  a problem  to  restore  f if 


cost  to  fix  f. 


'i  < 


B 


since  to  do  so  would  Increase  denominator  more  than  numerator  in  (4) . 

This  could  be  avoided  if  the  "cost  to  the  user"  of  not  having  f^  repaired 
were  reflected  in  some  recurring  cummulatlve  fashion  - perhaps  where 
t^  is  time  left  unrepaired. 


412 


H 


n 


5 . 2 Development  Quality 

The  quality  measure  for  operating  software  may  have  an  analog  for  develop- 
ing software  such  that  a quality  threshhold  might  be  established  as  an 
acceptance  criterion.  The  previously  described  measure  can  be  analyzed 
In  development  phase  terms  to  establish  a measure  for  that  phase.  Con- 
sider, for  example,  substituting  the  following: 


successful  tests 
total  tests 


and. 


. repair  cost 
development  budget 

and  given  a threshhold  of  0.95,  41  tests,  one  failure,  one  unit  of  cost 
fix,  £md  a 400  unit  budget: 


41 


0.974 


1 + 


400 


As  In  the  operating  quality  case,  the  difficulty  will  be  In  determining 
A rather  than  cost  accounting  M.  The  difficulty  Is  In  determining  A during 
reviews  or  during  debugging  under  Individual  programmer  control.  Since 
support  libraries  can  count  segments  and  segment  updates  automatically 
the  following  h3n>ochesls  Is  offered: 


A - 


segments 


segments  + updates 


updates  > segments  changed  as  a result  of  review  or  test 


The  hypothesis  Is  dependent  on  requirement,  design,  and  code  segments 
being  small  enough  to  be  complete  on  original  entry  to  the  library. 


I 

I 
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The  ratio  of  C/B  Is  a poor  choice  since  B Includes  development  work  and 
a large  development  effort  changes  Q drastically  even  though  the  program 
Q Is  supposed  to  measure  remains  unchanged.  Perhaps  better  would  be 
something  like 


where  Zc^  Is  the  actual  costs  of  repairs  made  and  C 
Is  the  cost  of  the  maintenance  staff.  C - Zc^  repre- 
sents the  breakage  of  having  to  keep  required  skills 
even  when  there  are  no  problems. 


— ^ 0 a pretty  good  program  (very  few  troubles)  but 

high  maintenance  overhead. 

— ^ -►  1 enough  troubles  to  totally  consume  budget  - 

little  breakage  but  perhaps  buggy  program. 

Maybe  quality  really  means  cost-effectiveness  and  we  are  Interested  In 
measures  like: 

o max  {A}  for  fixed  C 
o fixed  A for  min  {C} 

o rate  of  change  ^ 

dC 


The  author  wishes  to  express  his  gratitude  to  W.  C.  Cave  for  suggesting 
a quality  measure  for  managers  which  stimulated  this  note  and  part  of 
the  next.  D.  L.  Walker  made  substantial  contributions  to  the  note. 
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6.  Data 


The  author  has  one  firm  conviction:  software  quality  data  must  be  collected 
and  published  to  support  continuing  research.  Furthermore,  the  data 
must  come  from  real-world  software  systems  rather  tb'-i  small  experiments. 

The  key  questions  are  what  data  and  whether  that  d can  be  collected 
from  users,  developers,  malntalners,  and  operators. 

One  recommendation  of  this  workshop  should  be  the  Army  sponsor  a working 
group  to  Identify  the  reliability  and  quality  data  to  be  collected  and 
collect  these  data  through  their  In-house  and  contracted  programs. 

The  composition  of  the  working  group  should  Include  an  equal  of  re- 
searchers and  managers.  The  mix  will  yield  both  direct  and  indirect 
results.  The  possibilities  of  a researcher-manager  working  group  can 
be  envisioned  by  creating  dialog  for  the  quality  measure  note.  The 
thought  flow  in  the  note  is  as  follows: 


I 


fi 


o appeal:  simple.  Interesting  components 
o comparative  use 

o ambiguity  of  functlon/problem  priority 
o Implementation  difficulty 
o cost  accounting  Insight 
o manipulation 
o cost-to-fix  Insight 
o extensible  to  other  phases 
o implementation  difficulty 
o library  accounting  Insight 
o technical  dependency 
o cost/budget  difficulty 
o cost  effectiveness  Insight 
o other  candidates 
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A STJMMARY  OF  SOFTWARE  RELIABILITY  MODELS  AND  MEASUREMENT 


!mCARTIN  l.  shqoman 

0«p«rtxn«nt  of  Elactrieal  Eagiaaoriag,  Olrixion  of  Computor  Scicaco 


Polytechflic  lutitot*  of  York 
Brooidya»  Naw  Torfc  11201 


Wlth^  tha  adwaac  oi  larf  sopfaiaticatad  hardarar*-’ 
aoftwara  ayataraa  daralopad  is  tha  19M'a,  tba  prob- 
lan  ai  compmar  ayatam  raUafaility  ha»  aaargad. 

Tha  raliabiiity  of  eonapatar  hardirara  caa  ba  modalad 
la  nrach  tha  aaaaa  way  aa  othar  daaicaa  oaiaf  eoaraa» 
tifmal  railaMHty  tfaaaay;  howaaar,  coaipmar  aoffc» 
wara  arrora  raqoiza  a diffaraat  approaciu  Tha 
papar  bafiaa  by  daaexibiac  tha  typaa  aal  canaaa  of 
softarara  arrora  aad  proaidaa  worldxip  daflaitioaa  of 
softarara  arrora  aad  softarara  raliaUUty.  Soma  at 
dm  baaic  data  oa  fraqoaaey  of  oecnrraaca  of  arrora 
la  thaa  dtaeaaaadk  T^  papaa  tfaaa  soaaiarlsaa  aad 
rafaraaesa  soma  of  tha  softaara  raltafaillty  modala 
wMch  hara  baaa  propoaad  aad  eoacaatrataa  oa  oaa 
daralopad  by  tha  aathor.  Oaa  of  tha  probafaiUstte 
modala,  (tha  macro  modal),  pradleta  rallabillty 
baaad  oa  tha  Initial  laanbar  of  arrora  la  a program, 
tha  Tmmbar  ramorad,  aad  tha  aambar  ramaiaiag  ia 
tha  prograat.  Tha  modal  eoaotaats  ara  eaicolatod 
from  ^aratiaaai  taac  data  takaa  oa  tha  softarara  par* 
foraaaca,  Tha  othar,  tha  miero  modal,  foeaaaa  oa 
tha  patha  ia  tha  program,  thair  fraqoaaey  aad  dma 
of  rrararaal,  aad  tha  arror  rata  thma  paths. 

1.0  nmiopccTiow 

1. 1 Tha  Aaa  of  Larga  Comootara 

Tha  flrat  qaasttoa  dmt  oaa  hoars  ahaa  dta  tarm 
softaara  roUabiUty  is  manrionaii  ia  dfscaasioa  is, 
afaat  is  that?  As  tha  digital  eompator  eoatiaass  to 
parrada  mora  aad  mors  of  our  modara  taehaology, 
aa  raly  oa  its  ootpot  mora  aad  mora  for  eoatrol. 
data  raeordiag,  aoalysis,  aad  daclsioa  maidBg. 

Thaa,  ths  sisa  aad  complasity  of  tha  raqoirad  tasks, 
tha  eoatpotar  hardaaro,  aad  tha  compatsr  softaara 
baa  drastieally  iacraasad  ia  ths  last  thros  daeados. 

Ia  additioa,  tha  costa  of  softaara'  bars  groan  to  as— 
troaomical  proportioaa.  (Raeaat  astimatas  on  ths 
coat  of  softaara  to  ths  aatira  (X.  S.  aeoaomy  raags 
from  billion. ) With  snch  hog  a slaa  aad 

complasity,  it  is  rirtasUy  hapossibls  to  daflaitlToly 
spacify  dm  problam  (aithont  arror),  to  maka  &il> 
ora  fraa  eompntar  hardaara,  aad  to  ramora  all 
arrora  from  ths  softarara. 

Along  with  this  groath  has  coma  a raalisatiott 
that  ths  largast  atfort  ia  daralopiag  softaara  is  duo 
to  softaara  latagratioa,  tash  corractioa,  ratast, 
opatatioaal  ralaasa,  corractioa  aad  raralaaaa.  Ae- 
taal  arldag  of  tha  first  sac  of  coda  is  a small  task 
ia  comparisoB.  Aa  asaa  mora  eompalliag  obsarra- 
tioB  is  that  coaspatars  ara  iaeraasiagly  baiag  osad 
as  tha  haart  of  complaa  raal>cima  systams  sock  as 
sir  traffic  eoatrol,  ▼ahdcls  eoatrol,  spaca  systams. 


sad  military  systams.  Systam  rallability  is  ths 
moat  importaac  parfortBaaca  maasnra  ia  sach  sys— 
tarns. 

1.2  Sqms_ComgBtar  r»ilit*a  Data 

Ws  aiay  shad  huthar  light  oa  ths  distribotiaa 
batwaan  hardaara  sad  softaara  errors  by  eoasid— 
eriag  soma  sctoal  data.  Tha  data  girea  la  Tabla 
1^1  lists  hardaara  sad  softarara  fsilara  rataa  for 
a typical  raal-dma  data  scqnisitiaB  systam.  Tha 
data  rapraaaata  9 moatha  of  oparattoa  totaling 
1701  hoars.  laspactioa  of  tha  data  shoora  that  48% 
of  ths  fsilaras  aars  dna  to  softaarok  This  is  a 
tesrtiiag  figors  if  oas  rsaiisss  that  most 

caaspotar  projseta  ssdaaats  sad  prsdict  tha  hard- 
aara raliahility,  sad  ass  rsdoadsacy  tschaiqass 
sad  high  raliahility  parts  to  imprors  ths  hardaara 
raliahility,  tha  softaara  la  laft  to  ths  skill  sad 
hard  aork  of  ths  prograauaiag  taam  aids  ao  quaa— 
dtatlrs  asasasmaat  of  dasiga  prograsa.  Wa  may 
obtais  a typical  aatfanata  of  ahat  typa  of  systam  ra— 
liability  caa  bs  ofataiaad  ia  practiea.  Assamiag  a 
siatpls  faihiTs  modal,  as  may  compnta  ths  msaa- 
dms  to  failars,  MTT7  as  ths  raciprocal  of  ths 
failarsrats.  Thas.  ths  bCTTF  « 30  hoars  for  ths 
data  la  Tsbls  l-l.  Ia  othsr  aorda  sboor  oos  fail- 
ars pm  day  aill  oecar  if  ths  aqoipmsm  is  ossd  24 
hoars/ day  sad  oaa  fsiloro  avory  thras  daya  if  oaad 
a hoors/day.  (Additiaaal  data  la  giraa  ia  Raf.  2.  } 

1.3  Hardaara  rs.  Softaara 

la  stodyiag  tha  eaoaaa  at  failura,  aa  idaatify 
fsilnraa  by  tha  any  thay  sffaet  tha  systam  (moda) 
sad  by  thair  esasaa  (machsirisms).  la  hardasra 
aa  spaak  a£  "early  fsilores*  dna  to  poor  parts, 
dsmsga  ia  traasportstioa.  sad  dssi^  srrers; 
"middla  of  Ufa  fsilnraa*  eaasad  by  s high  load 
straaa  aaeaadiag  a part's  straagth;  sad  "asar  oat 
failaraa*  aUeh  occor  sftsr  mock  oaa  daa  to  ma* 
rhanirsl,  alactrieal,  or  chamical  ehaagaa  ralatad 
to  sgs  sad  aaar.  la  tha  cssa  of  softaara.  almost 
all  tha  arrors  ara  daalga  arrora.  (Thsra  may  ba 
soma  arrors  daa  to  a peer  disk.  taps,  or  paachsd 
card  soores  of  tha  softaara  or  arrors  ia  iastalla* 
tioB.)  Also  I.shfnan  aad  Bslady  la  Rsf.  3 sad  4 
dsseribs  ths  eocapoaad  sffset  of  ehaages  la  speel- 
fieatioas  sad  addad  fsstarsa  as  Groath  Oyasmics, 
ahich  rssalts  ia  so  iaersssiag  fsilors  raus  rsry 
mash  liks  assr  oat.  Thaa,  oat  at  ths  many  modsls 
ossd  ia  hsrdaars  failars  saslyais,  only  ths  oass 
dsscribiag  dasiga  arrors  fiad  aids  ass  la  softaara. 

2.0  pgrpirnoN  or  a softwarr  error 

Ths  fpUeaiag  dsfiaidea  of  s sofnrare  arror  is 
propessd:*Oas  or  more  eoftasre  errors  exist  la  a 
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•yttam  if  a •oftorare  chaag*  is  rsquirad  to  corroct 
systaza  parformajaca  so  as  to  mast  spacificatioiis.* 
bharaar  ia  tha  abova  dafimtioas  aad  discussioa  is 
tha  assazaptloa  that  arrors  can  ba  aad  ara  datactad 
and  racordad.  Tha  datactioa  of  arrors  can  ba  sf> 
factad  by  monitoring  tha  syatam  (or  simalatad  sys- 
tam)  partorxnanca  or  by  raading  tha  coda.  Forthar* 
mora,  it  is  assumad  that  each  arror  is  sufficiantly 
wall  invastigatad  so  that  it  can  ba  classified  as 
hardware,  software,  operator,  or  unknown,  and 
that  tha  unknown  category  is  small,  say  lass  than 
20%. 


3.0  DEFPIITIOW  OF  SOFTWARE  REUABHITT 


3. 1 Oafinitioo 


A dafinitioa  for  softwaza  reliability  is  given  be- 
low in  keeping  with  tha  factors  discussed  above. 

This  definition  is  a slight  modification  of  one  given 
by  Hassat^  "Softaraza  reliability  is  defined  as  die 
probability  that  a given  softwaza  program  oparatas 
for  some  time  period,  without  software  arror,  on 
the  machine  for  which  it  was  designed  given  that  it 
is  used  within  design  limits.  * Once  we  have  re- 
lated reliability  to  a probability,  as  in  the  above 
definition,  the  mathematical  basis  of  the  measure 
is  well  founded;.  Of  course,  dxe  problems  in  Intarp- 
ratinc  terms  such  as  arror  and  design  limits  still 
exists. 


4.0  ERROR  DATA  AND  MODELS 


Error  Data 


As  is  the  case  with  all  studies,  data  is  difficult 
aad  generally  costly  to  obtain.  Good  aad  complete 
recces  ace  not  kept  in  most  sitnatioas.  Record 
keeping  is  ganaraily  better  in  the  case  of  large 
military  aad  space  programs  and  after  a release  of 
a large  commereial  operatiag  system.  Although  a 
fairly  large  amount  of  such  ^ta  exists,  military 
secrecy  tad.  industrial  proprietary  policies  inhiUt 
its  pnblleatioa  in  many  cases.  Some  ai  this  data 
which  has  bean  published  appears  in  Rafs.  2.  5. 14, 
(Also  sea  paper  by  Shoomaa  aad  Bolsky  in  Raf.  6 ). 


Assume  that  a typical  operatiag  system  for  a 
large  computer  is  imdargoiag  contiisial  develop- 
ment aad  that  new  features  si^  capabilitias  are  be- 
ing added  The  manufacturer's  development  group 
d«nls  with  a continually  changing  product,  but  ex- 
ternal versions  (generally  called  releases)  are  only 
made  available  periodically,  say  every  6 months. 
Although  the  manufacturer  tries  to  thoroughly  test 
each  release,  the  exercising  of  the  program  by  a 
fair  proportion  of  the  large  aad  diverse  user  com- 
munity is  more  comprehensive  than  any  test  he  can 
devise.  Conseguently,  soon  after  release  of  a new 
version,  the  number  cd  errors  found  per  month  (er- 
ror rate)  rises  rapidly  to  a peak.  As  these  are  di- 
agnosed aad  corrected,  the  number  of  residaal  er- 
rors decreases  aad  the  error  rate  begins  to  de- 
crease. When  a new  release  is  distributed,  this 
behavior  is  repeated  Detailed  data  on  the  normal- 
ised number  of  errors  since  release  for  three  dif- 
ferent supervisory  systems  (operatiag  systems)  is 
given  in  Fig.  4-1.  Note  that  the  horlsoa^  axis 
units  are  months  of  debugging  r.  In  this  case  r is 
identical  with  operatiag  time,  t;  however,  this  is 
net  always  the  case. 


Note  that  the  shapes  depicted  in  Fig.  4- la,  b,  c 
vary.  If  we  assume  that  the  number  of  remaining 
arrors  decreases  monotonically  and  that  the  er- 
ror discovery  rate  is  proportional  to  number  of 
remaiaiag  errors,  ejqzonential  decay  is  obtained. 
This  ei^l^as  in  a gross  way  the  "tail”  of  the 
curves.  Tha  behavior  may  be  due  to  the 

fact  that  initially  only  a few  installations  are  us- 
ing the  new  release,  and  it  is  not  until  a few 
months  later  that  a sixeable  proportion  of  users 
have  instituted  this  software.  Thus,  it  might  be 
more  appropriate  to  let  r represent  a more  gen- 
eral resource  variable  such  as  user-months,  or 
to  serve  as  a more  realistic  horizontal  scale 
parameter.  (See  Ref.  3 for  a more  detailed  dis- 
cussion of  ttose  curve  shapes, ) 


4.  2 Error  Model 


Referring  to  the  data  discussed  in  die  previous 
sectioa  we  see  that  although  the  curve  shapes  dif- 
fer, the  vertical  and  horizontal  scales  are  simi- 
lar. Based  on  this  result  we  can  proceed  to 
formulate  a general  error  model  using  the  num- 
ber of  machine  language  instructioiia  as  a 
normalizing  factor.  (Sec.  4.  3 discusses  the  possi- 
bility that  the  total  number  of  operators  aad  oper- 
ands is  a better  normalizing  factor. ) 


Basically,  die  error  model  used  in  this  paper 
assumes  that  the  total  number  of  errors  in  ihs 
program  is  fixed  aad  that  if  we  record  the  cumu- 
lative number  of  errors  corrected  during  debug- 
ging, then  the  difference  represents  the  remain- 
ing err  ore.  The  foUmring  section  on  reliability 
models  will  relate  the  probability  of  encountering 
a softsare  bug  to  the  number  of  resldnal  bugs. 


The  normalized  error  rate  Is  defined  as 


p(r)  V errors/ total  number  of  iastructioas/ 

mondi  of  debugging  time.  (4-1) 

Thus.  Figs.  4-1  a.b.  c are  plots  of  p(r)  vs.  r. 


Since  we  are  interested  in  the  total  number  of 
errors  removed,  we  will  define  a cumulative  er- 
ror curve,  e(r),  which  is  the  area  under  the  p(t) 
curve: 


e(r)  « f p(z)dx  ■ cumulative  errors/total 
o number  of  iastructioas 

(4-2) 

aad  p(r)  is  of  course  the  slope  of  the  c(r)  curve: 

0(t)  * dc(T)/dT  (4-3) 


A curve  of  the  cumulative  error  data  for  the 
supervisory  system  A of  Fig.  4-1  is  shown  in 
Fig.  4-2.  U similar  curves  for  c(t)  were  drawn 
for  the  ether  examples  of  Figs.  4-1  all  would 
start  at  zero,  increase  slowly,  then  move  rapidly, 
aad  finally,  more  slowly  approaching  a slowly  in- 
creasing or  zero  rate.  Because  of  this  similarity 
in  behavior,  a cumulative  curve  such  as  c(t)  Is 
not  too  useful  to  depict  differences  in  behaviors: 
thus,  the  derivative  curve,  p(t),  is  more  useful 
for  this  purpose.  Both  curves  are  needed  for  a 
detailed  study. 


If  we  assume  that  the  total  number  of  erroirs 
in  the  program  Ey  is  constant  aad  that  the  pro- 
gram congas  ly  total  instructions  aad  that  no 
new  errors  are  added  during  debugging,  dtea  the 


aayrnptote  which  the  G(r)  curves  approach  is 
U we  assume  that  all  detected  errors  are  corrected 
errors,  then  by  inspectioD  of  Fig.  4-2,  we  can 
write  an  expression  for  the  number  of  residual  er- 
rors: 


e,{T)  » - €^(t) 


We  assume  that  in  any  sizeable  program  it  is  im- 
possible to  remove  all  errors,  so:  e (r)  < £'p/lT 
and  er(T)  > 0.  Also  since  we  asaume*^tbat  moat 
programs  eventually  reach  a reasonable  debugged 
state,  we  may  assume  that  for  large  r,  cirllx/ET 
is  smaU. 

hi  order  to  test  the  hypothesis  that  the  normal- 
ized behaviors  of  er(T)  and  p(t)  hold  for  a wide  va- 
riety of  program  sizes  we  make  the  following  com- 
parisons with  the  data  in  Fig.  4-1:  (1)  In  order  to 
test  the  hypothesis  that  the  normaliz^  number  of 
errors  Ey/lT  i*  somewhat  constant  for  a variety 
of  programs,  we  compute  the  ratio  and  compare 
the  results.  (2)  An  allied  hypothesis  is  that  debug- 
ging proceeds  at  a roughly  similar  average  rate  po 
over  an  entire  project.  The  results  are  given  in 
Table  4-1.  The  value  of  E^/^T  varies  about  the 
average  by  M8%  and  -31%  and  diat  of  p^  varies 
about  the  average  by  •t-75%  and  -31%.  Mote  that  all 
these  programs  are  about  1/4  million  machine 
language  statements  in  size.  We  now  present  simi- 
lar data  for  small  programs  in  Table  4-2.  In  this 
case  both  the  values  of  Fy/Ix  and  pg  vary  about 
the  average  by  >79%  and  -36%. 

The  data  in  Table  4-2  includes  data  taken  during 
module  test  as  weU  as  during  integration  testing 
and  as  might  be  expected  (because  of  the  two  phases 
being  lumped),  the  average  value  of  E^/^T  the 
small  programs  data  is  2. 15  times  larger  than  the 
large  program  data  whereas  the  value  of  p^  is  1. 53 
times  larger.  Drawing  these  various  facts  together 
allows  us  to  state  that  within  a factor  of  perhaps  2, 
the  values  of  Ey/lT  Po  *eem  to  be  constant  for 
a wide  variety  of  programs. 

One  Airther  comment  is  in  order  before  we  leave 
the  subject  of  error  models.  Some  experienced  pro- 
grammers have  challenged  the  assumption  that  no 
new  errors  are  generated  during  debugging.  A 
newly  devised  model,  formulated  by  this  author  and 
his  co-werkars  describes  error  generation  and 
is  discussed  and  developed  in  Reference  7. 


4.  3 Program  Ls 


and  Program  Complexit 


There  are  many  similaritiea  between  natural 
and  computer  languages,  and  we  will  make  use  of 
the  analogies  between  nouns  and  verba  and  oper- 
ands and  operators  to  obtain  better  length  measures 
of  a program  based  on  Zlpf's  law  applied  to  pro- 
grams. 14,15 

Zipf  studied  the  relatioaship  between  relative 
frequency  of  occurrence  f^,  (actually  the  number  of 
occurrences  n^  of  rank  r words  in  a sample  of 
size  n),  and  rank  r for  words  from  English,  Chinese 
and  the  Latin  of  Platus^. 

Carefttl  study  at  Zipfs  data  and  that  of  others 
shows  that  vs.  r plots  as  a straight  line  on  log- 
log  paper,  with  a unity  slope,  thus  we  arrive  at 


the  simple  relationsh^  called  Zipf's  law, 
n 

f^.  r a ^ . r » c (4-5) 

The  constant  c can  be  interpreted  as  the  relative 
frequency  of  the  rank  1 word  type,  (also  the  inter- 
cept with  the  r a 1 line). 

Solving  for  n^  and  summing  both  sides  of  Eq. 
4-S  for  the  t different  word  types  we  obtain  * 
t t 

n a)^  * cn^  ~ (4-6) 

r«l  r*! 

The  summation  of  the  series  l/r  is  given  by 

£i. 0.5772, ,4-71 
ral 

Substitution  from  £q.  (4-7)  into  Eq.  (4-6)  (retain- 
ing only  2 terms  for  modest  size  t)  yields 

a • cn  (0.  5772  + In  t)  (4-8) 

We  can  eliminate  the  constant  c from  Eq. 

(4-8)  by  considering  the  behavior  of  Zipf's  law  for 
the  smsullest  rank  which  is  where  r.„^«.at(eg.  if 
there  are  100  types,  then  the  largest  rank  is  ob- 
viously 100).  In  most  cases  the  rarest  type 
(largest  rank)  will  occur  only  once,  thus,n-  si. 
Substituting  these  values  yields,  cat/ n,  ^nx 

which  when  combined  with  Eq.  (4-8)  gives 

a a t(0.  5772  > In  t)  (4-9) 

The  results  to  date  have  shown  that  both  oper- 
ators, operands,  and  the  sum  of  operators  plus 
operands  flt  Zipf's  law  fairly  well  >.  Thus,  in- 
stead of  using  I.p  as  a norm^zing  factor  in  Eq. 
(4-1)  to  (4-4)  one  could  use  total  word  length  n. 

4.  4 Estimation  of  Program  Length  Early  in 


A method  of  initially  estimatiag  program 
length^  is  to  assume  the  analyst  initially  has  a 
complete  description  of  the  problem  and  that  a 
partial  analysis  and  choice  ^ key  algorithms  has 
been  made.  An  elementary  approach  might  be  to 
estimate  the  token  size  by  (1)  Estimating  the  num- 
ber of  operator  typer  which  will  be  used  in  the 
language  by  the  assigned  programmers.  (2)  Esti- 
mating the  number  of  input  variables,  output  vari- 
ables, intermediate  variables,  and  constants 
needed  (3)  Sum  the  estimates  of  step  (1)  and  (2) 
and  substitute  for  t in  Eq.  (4-9). 

5.0  RELIABILITY  MODELS 
5.  1 Basic  Assumptions 

We  assume  that  operational  software  errors 
occur  due  to  the  occasional  traversing  of  a por- 
tion of  the  program  in  which  a hidden  software 
bug  is  lurking.  We  begin  by  writing  an  expression 
for  the  probability  that  a bug  is  encountered  in  the 
time  interval  At  ^ter  t successful  hours  of  oper- 
ation. This  nmst  be  proportional  to  the  proba- 
bility that  any  randomly  chosen  instruction  con- 
tains a bug,  i.  e. , the  fractional  number  of  re- 
maining  bugs  c(t). 

*Most  frequent  word  ^e  is  rank  U second  most 
frequent  is  rank  2,’ 'least  frequent  is  rank  t. 
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From  a study  of  basic  probability  and  relia  - 
bility  theory,  lO  we  learn  that  the  probability  of 
failure  in  time  interval  t to  t fit  given  that  no 
failures  have  occurred  up  till  time  At  is  propor- 
tional to  the  failure  rate  (hazard  function  z(t},  and 
the  reliability  is  related  to  the  failure  rate 


P(t  < tj  < £ + At  I tj  > ^)  a z(t)  At 


Ker(T)At 

where  t . a operating  time  to  failure,  (occurrence 
of  a soTlware  error) 

P(t  > t)  ■ probability  of  failure 

in  interval  At,  given 

Kaanarbitrary  constant  ““  P'*^*"* 

-L*(x)dx 


R(t)=e 


(5-2) 


Note  that  in  Eq.  5-1  two  time  variables  ^pear: 
first  there  is  t the  operating  time  in  hours  of  the 
system  and  second  there  is  r the  debugging  time  in 
months  (or  more  generally,  the  debugging  resource 
variable).  Once  the  assumptions  in  Eq.  5-1  have 
been  made,  the  reliability  and  mean  time  to  failure 
functions  follow  directly. 

5.  2 Reliability  Model 

By  combining  Eqs.  (5-1),  (5-2),  and(4-4)  and  as- 
suming that  K and  c^ir)  are  independent  of  operat- 
ing time  t,  we  obtain  for  the  reliability  function 

»e  ao^«e^  J 


R(t) 


(5-3) 


Basically  the  above  equation  states  that  the 
probability  of  successful  operation  without  soft- 
ware bugs  is  an  exponentisil  function  of  operating 
time.  When  the  system  is  first  turned  on,  t^O  and 
Ral.  As  operating  time  increases  the  reliability 
monotonicidly  decreases  as  shown  in  Fig.  5-1.  We 
depict  the  reliability  function  for  three  values  of  de- 
bugging time,  Tg  <Tj  <^7  . From  this  curve  we 
may  make  vs^ws  predictions  about  the  system  re- 
liability. F or  example,  looking  along  the  vertical 
lins  tel/v  we  may  state: 

1.  If  we  spend  tq  hours  of  debugging,  then 

R(1/y)  » 0.35 

2.  If  we  spend  ri  hours  of  debugging,  then 

R(l/v)  a 0.50 

3.  If  we  spend  rv  hours  of  debugging,  then 

R(l/v)  » 0.75 

5.3  MTTF  Model 

A simpler  way  to  summarise  the  results  of  the 
reliability  model  is  to  compute  the  mean  time  to 
(software)  failure,  MTTF  by  using  the  formula 


MTTF  a /^R(t)dt.uj^j  « 


1 


* e^(T)| 


If  we  let  o(')  be  modeled  by  a constant  rate  of 
error  correction  o (see  Ref.  3 for  other  models), 
then  ® 

1 _ 1 

8(l*aT) 


MTTF 


(5-5) 
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'T  Po^T 

where  9 » »—  K and  a » ■ 


In  Fig.  5-2,  9 x MTTF  is  plotted  vs.  at.  We 
see  that  the  most  improvement  in  MTTF  occurs 
during  the  last  1/4  of  the  debugging. 

Other  similar  models  have  been  proposed  anri 
the  reader  is  referred  to  the  literature:  Jelinski 
and  Moraada  ‘^.Schick  and  Wolverton 

5.  4 Experimental  Reliability  Data 

If  we  had  just  deployed  a large  hardware-soft- 
ware  system  for  field  use,  we  could  monitor  its 
reliability  by  carefully  recording  the  operating 
time  and  documenting  each  failure  in  detail.  Thus, 
we  could  obtain  the  times  between  failure.  In- 
vestigation of  each  failure  should  allow  ope  to 
classify  all  failures  as  hardware,  software,  op- 
erator, or  unknown.  If  we  segregate  the  software 
times  between  failure  and  plot  their  average  week 
by  week,  we  will  have  a quantitative  measure  of 
operational  software  reliability.  We  would  expect 
the  operational  MTTF  to  increase  for  the  first 
month  (year,  in  some  cases)  or  so  as  software 
bugs  detected  in  service  are  removed,  then  gradu- 
ally to  level  off  to  a relatively  constant  value.  This 
is.  of  course,  an  after-the-fact  evaluation  of  the 
software  design  and  does  not  allow  one  to  measure 
progress  and/or  need  for  improvement  of  the  soft- 
ware design  while  it  is  under  development. ' 

The  earliest  stage  at  which  an  entire  system 
can  be  functionally  tested  is  during  system  inte- 
gration using  the  system  exerciser  (fiinctioivsl 
test)  program.  If  this  tost  is  performed  at  the  be- 
ginning of  system  integration,  the  result  will  be  a 
succession  of  very  short  runs  and  immediate 
crashes.  Most  software  test  personnel  would  in- 
stinctively comment  that  this  is  as  expected  since 
the  system  is  stiU  in  *poor  shape*  and  such  a test 
should  be  delayed  until  the  end  when  the  system  is 
in  'good  shape”.  A bit  of  reflection  leads  one  to 
the  conclusion  that  it  is  just  this  frequent  crashing 
which  leads  to  a quantitative  assessment  of  the 
poor  initial  reliability. 

W e now  focus  on  the  test  data  and  how  it  should 
be  analyzed.  The  necessary  information  which 
must  be  recorded  for  each  run  of  the  system  test 
program  is  how  long  the  test  ran,  whether  an  er- 
ror occurred,  and  if  the  error  is  a software  error. 
Sufficient  dumps  and  other  documentation  must  be 
recorded  for  subsequent  analysis  in  order  to  seg- 
regate errors  into  hardware,  software,  operator, 
etc. , errors.  Each  of  the  r successful  runs  repre- 
sent Ti,  TZ, . . . Tf  hours  of  success.  If  there  are 
a total  runs,  than  each  (a-r)ex  unsuccessful  run 


(5-4)  represents  t^,  h, . . . t_.^  successful  run  hours  be- 
fore failure.  Tne  totu  number  of  successful  run 


hours  H is  given  by 


n-r 


H 


‘1 


(5-6) 


i>l  i«l 

Assuming  that  the  failure  rate  is  constant,  we 
denote  it  by  X and  compute  it  as  the  number  of 
failures  per  hour,  along  with  its  reciprocal  which 


is  the  MTTF  (for  a constant  failure  rate,  c.  f. 

Eqs.  (S-3)  and  (5-4)) 

Failure  Rate  “ X “ •g  (5-7) 

1 H 

MTTF»i»^  (5-8) 

Since  we  are  interested  in  software  failures,  we 
assume  that  the  outputs  as  well  as  dumps  are 
carefully  investigated  for  the  x failures,  and  the 
failures  are  divided  into  hardware  failures, 

X,  software  failures.  Xg  operator  failures,  and  x^ 
unknown  failures.  (Hop^ully  x /x  will  be  small. 
Then  the  software  failure  rate  and  MTTF  are  de- 
fined by  Eqs.  (5-7)  and  (SS)  with  x substitutsd  for 
X . 

Baaed  upon  the  results  of  Sec.  5.  3 and  5.  4 we 
can  calculate  the  model  constants.  We  assume  a 
known  program  size  and  careful  collection  of  er- 
ror data,  then  and  eg(r)  are  known  values  and 
only  the  constants  K and  Eq-  remain  to  be  deter- 
mined. These  two  unknowns  K and  E-p  can  be 
evaluated  by  running  a functional  teat  after  two 
different  debugging  times,  < ‘’I?  chosen  so  that 
e (r.)  < C (r,).  We  then  equate  Cqs.  (5-4)  and 
(f-8r  at  times  Tj  and 


6.0  MICRO  RELIABIUTY  MQDEUt 
6.  1 Introduction 

The  software  reliability  prediction  model  of 
Sec.  5 concentrated  on  the  bulk  (macro)  aspects  of 
a program.  This  secdon  describes  a newly  de- 
veloped micro  model  ^^which  is  based  on  program 
structure. 

It  is  assumed  that  the  program  has  been  written 
in  structured  or  modular  form  so  that  decomposi- 
tion into  its  constituent  parts  is  simple.  Further, 
we  assume  that  via  analysis  of  the  program  the  de- 
composition can  be  related  to  several  paths  or 
other  functional  structures  within  the  program. 

The  model  is  constructed  based  upon  the  fre- 
quencies with  which  each  of  the  j paths  are  run. 

(f.),  the  running  time  of  each  path,  (t,),  and  the 
probability  of  error  along  each  path,  Iq.). 

6.  2 Development  of  the  Model 

The  model  is  developed  by  calculating  an  ex- 
pression for  the  number  of  failures,  n^'  , experi- 
enced in  14  tests  of  the  system,  and  the  number  of 
test  hours  H.  (see  Ref.l6).Then  the  system  failure 
rate  z^  is  computed  from 


■ ®c<^l>l 


(5-9) 


1 


(5-10) 


Taking  the  ratio  of  Eq.  (5-9)  to  (5-10)  and  using  Eq. 
(5-8)  yields 


(5-11) 


A 

Once  E .p  (las  been  computed  from  Eq.  (5-11),  we 
obtain  K by  substituting  Eq.  (5-11)  into  (5-9) 
which  yields 


K « X,^/[(ET/I.r)  - CcCtj)]  (5-12) 

The  *bats*  above  Ex  sad  K in  Eqs.  (5-11)  and 
(5-12)  denote  estimates  of  the  parameter.  Further 
discussions  of  this  parameter  estimation  technique, 
the  more  powerful  maximxim  likelihood  estimatioa 
technique,  and  accuracy  questions  appear  in  Ref.  13. 

5.  6 Verification  of  the  Model 

Two  papers  have  appeared  in  the  literature  which 
apply  the  above  model  or  extensions  of  it  to  practi- 
cal data.  The  paper  by  Miyamoto  in  Ref.  6 reports 
results  very  similar  to  Fig.  5.  2.  In  a more  exten- 
sive study  oi  four  program  developments  Musal^ 
reported  good  agreement  between  MTTF  estimated 
during  development  (c.  f.  Lqs.  (5-5),  (5-1 1),  (5-12) 
and  subsequent  operational  measurements. 


The  term  N appears  in  numerator  and  denominator 
and  Eq.  (6-1)  yields  in  the  limit 


1 


j-i 


(6-2) 


Equation  (6-2)  can  be  substituted  into  Eq.  (5-2)  to 

obtain  a reliability  modeL  Measurement  of  the 

model  parameters  is  discussed  in  Ref.  16. 
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TABLE  4-1 
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SOFTWARE  RELIABILITY  MEASUREMENT* 

John  D.  Musa 
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Whippany,  New  Jersey  07981 


ABSTRACT 

The  quantification  of  software  reliability  is  needed  for  the  system  engineering 
of  products  involving  computer  programs  and  the  scheduling  and  monitoring  of 
software  development  It  is  also  valuable  for  the  comparative  evaluation  of  the 
effectiveness  of  various  design,  coding,  testing,  and  documentation  techniques.  This 
paper  outlines  a theory  of  software  reliability  based  on  execution  or  CPU  time,  and  a 
concomitant  model  of  the  testing  and  debugging  process  that  permits  execution  time 
to  be  related  to  calendar  time.  The  estimation  of  parameters  of  the  model  is  dis- 
cussed. Application  of  the  theory  in  scheduling  and  monitoring  software  projects  is 
described,  and  data  taken  from  several  actual  projects  are  presented. 


I.  INTRODUCTION 

Three  important  current  needs  in  the  field 
of  software  engineering  have  been  instrumental 
in  stimulating  work  (and  more  particularly,  the 
author's  work)  in  developing  a way  to  measure 
software  reliability: 

(a)  an  urgent  requirement  to  qtuntify  relia- 
bility due  to  its  importance  as  a parame- 
ter in  the  engineering  of  software  sys- 
tems 

(b)  the  need  for  better  methods  of  establish- 
ing and  monitoring  schedules  for 
software  development  projects,  and 

(c)  the  necessity  of  evaluating  the 
effectiveness  of  different  software 
engineering  techniques. 

The  growing  trend  in  computing  toward 
interactive  dau  processing  systems,  both  time- 
sharing and  transaction-based,  has  resulted  in 
an  increasing  focus  on  the  reliability  of  such 
systems,  because  of  the  greater  operational  and 
cost  impacts  of  real  time  malfunctions.  For 
example,  consider  the  effects  of  breakdowns  of 
airline  reservations,  banking,  or  stock  broker- 


*  PrtMnted  at  the  Software  Life  Cycle  Manatement 
Workshop,  Airtie.  Virginia.  Augtist  22-2J.  1977. 


age  online  systems.  The  concurrent  trends 
toward  distributed  processing  and  networking 
have  increased  the  scope,  complexity,  and 
interdependence  of  computing  systems,  multi- 
plying the  risk  of  failure  and  therefore  further 
reinforcing  the  concern  over  reliability. 

Measurement  becomes  important  as  soon 
as  one  recognizes  that  in  software  as  in 
hardware  there  can  be  too  much  as  well  as  too 
little  reliability.  Improvement  of  reliability,  of 
coune,  costs  money,  and  usually  impacts 
development  schedules  and  system  perfor- 
mance (in  the  case  of  software,  through 
increased  memory,  processing  time,  and  peri- 
pherals requirements).  The  system  engineer 
and  the  manager  have  to  make  design  tradeoffs 
between  reliability,  performance,  cost, 
schedules  and  other  factors,  either  implicitly  or 
explicitly.  There  is  a pressing  need  for  good 
tools  that  will  permit  the  decisions  to  be  made 
txpUcitfy.  Such  tools  exist  in  the  case  of 
hardware,  but  there  has  not  been  a correspond- 
ing development  in  the  area  of  software.  How- 
ever, software  is  an  and  usually  the  essential 
element  in  the  systems  we  have  been  describ- 
ing. Consequently,  there  is  an  urgent  need  to 
find  ways  to  characterize  and  measure  software 
reliability  and  to  develop  methods  of  combining 
reliabilities  of  software  and  hardware  elements 
to  aaublish  system  reliability. 


• 1977,  Bell  Telephone  Laboratories,  Inc. 
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A second  issue  of  concern  in  the  dau  pro- 
cessins  community  is  how  to  schedule  a 
software  development  project  and  monitor  its 
progress.  This  task  becomes  both  most  critical 
[1]  and  most  dilTicult  during  the  latter  phases 
of  the  project  where  testing  and  removal  of 
errors  is  the  primary  activity.  The  time 
required  for  testing  is  mainly  dependent  on  the 
initial  reliability  of  the  design  and  code  and  the 
reliability  goal  to  be  attained.  Fred  Brooks  [2] 
points  out  the  uselessness  of  "man  month 
estimating,"  particularly  in  the  component  test 
and  system  test  phases,  because  of  the  sequen- 
tial nature  of  debugging.  He  also  emphasizes 
the  criticality  of  the  system  test  phase  to  a pro- 
ject, and  the  need  for  better  estimating  tech- 
niques. 

Testing  progress  has  been  generally  very 
difficult  to  evaluate.  Managers  have  used 
methods  such  as  intuition  of  designers  or  test 
team,  percentage  of  tests  completed,  and  the 
successful  execution  of  critical  functional  tests. 
None  of  these  have  been  completely  satisfac- 
tory and  some  have  been  most  unsatisfactory. 
On  the  other  hand,  reliability  (or  better,  present 
mean  time  to  failure  (MTTF)]  offers  substan- 
tial promise  of  being  a good  monitoring 
metric.* 

Third,  the  computing  field  has  seen  and 
continues  to  see  a plethora  of  new  software 
engineering  techniques:  requirements 

definition  approaches,  design  techniques,  cod- 
ing methods,  testing  techniques,  and  documen- 
ution  methodologies.  Unfonunately,  there  has 
been  little  quantiutive  evaluation  of  these  tech- 
niques. The  initial  enthusiasm  that  greeted 
such  innovations  (due  perhaps  to  the  general 
recognition  of  software  engineers  of  the  need  to 
develop  their  technology)  soured  and  turned  to 
skepticism  as  many  of  the  ideas  turned  out  to 
be  not  sufficiently  effective  to  justify  their  cost. 
The  inability  of  managers  and  practitioners  to 
differentiate  between  good  and  bad  new  tech- 
niques has  led  to  a general  resistance  to  change 

* In  wmt  cun  cherc  may  b«  • more  nauinl  uniL 
For  axunpit.  in  tranuction  proctaaini  syswms  wbtre 
ona  can  dafina  a “naodani’'  common  irantaction. 

Mci)  u a raaarvation  for  airtina  raurvaiion  lytwms. 
ona  may  wiali  to  talk  in  terms  of  raservaiiOM  pro* 
ceased  batwsan  failum  raibar  than  MTTF.  The 
conversion  is  easily  made. 
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that  is  counterproductive.  Finding  a way  to 
measure  reliability  offers  promise  of  esublish- 
ing  at  least  one  criterion  for  the  evaluation  of 
new  technology.  For  example,  you  would  then 
be  able  (in  an  experiment)  to  concretely  deter- 
mine the  increase  in  MTTF  at  the  surt  of  sys- 
tem test  that  results  from  using  structured  pro- 
gramming. You  could  compare  the  dollar  value 
of  this  benefit  with  the  increased  cost  required 
in  the  design  phase. 

The  objective  of  the  main  body  of  this 
paper  is  to  give  an  overview  of  a software  relia- 
bility theory  developed  by  the  author  and  its 
appUcation.  Additional  details  that  will  be  use- 
ful to  those  who  wish  to  apply  the  theory  are 
contained  in  the  Appendices.  A brief  glossary 
of  terms  is  provided  in  Appendix  A. 


II.  software  reliability  vs. 

HARDWARE  RELIABILITY 

Software  reliability  will  be  defined  like 
hardware  reliability;  i.e.,  it  is  the  probability  of 
failure-free  operation  in  a specified  environ- 
ment for  a specified  time.  A "failure"  is 
defined  as  an  unacceptable  departure  of  pro- 
gram operation  from  program  requirements. 

An  "error”  is  the  software  defect  that  causes 
the  failure.  The  concept  of  software  reliability 
differs  from  that  of  conventional  hardware  reli- 
ability in  that  failure  is  not  due  to  a wearing-out 
process.  Once  an  error  is  properly  fixed,  it  is. 
in  general,  fixed  for  all  time.  Failure  usually 
occun  only  when  a program  is  exposed  to  an 
environment  that  it  was  not  designed  or  tested 
for.  For  most  programs,  in  practice,  the  large 
number  of  possible  sutes  and  inputs  makes 
perfect  comprehension  of  the  program  require- 
ments, perfect  implemenution  and  complete 
testing  generally  impossible.  Thus,  software 
reliability  is  essentially  a measure  of  the 
confidence  we  have  in  the  design  and  its  ability 
to  function  properly  in  its  expected  environ- 
ment. Although  the  failure  mechanism  is 
different  for  software  than  hardware,  the  result 
for  the  system  is  the  same;  this  is  why  a theory  r ■ 

that  is  compatible  with  hardware  reliability  can 
be  developed. 
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111.  BASIC  CONCEPTS 

Jelinski  and  Moranda  [3]  developed  one 
of  the  earliest  software  reliability  models.  They 
postulate  an  error  detection  rate  that  is  propor* 
tional  to  the  number  of  errors  remaininf. 
Miyamoto  [4]  has  published  data  confirming 
this  relationship.  Shooman  [Sl.(6|  used  a simi* 
lar  model,  relating  error  detection  rate  also  to 
the  program  size  and  instruction  processing 
rate.  He  investigated  various  models  for  error 
correction  and  proposed  a two>point  parameter 
estimation  approach  Schneidewind  (71,  [81  took 
an  empirical  approach  and  suggested  a reliability 
prediction  scheme  based  on  fitting  failure  inter* 
vals  with  an  appropriate  reliability  function.  He 
has  developed  models  for  error  detection  and 
correction  processes  and  applied  maximum 
likelihood  estimation  to  the  determination  of 
the  parameters  of  these  processes  [101.  Little* 
wood  and  Verrall  [II],  [12]  developed  a very 
general  Bayesian  reliability  growth  model  for 
software  in  which  repair  actions  diminish  the 
failure  rate  in  a probabilistic  rather  than  deter* 
ministic  fashion. 

After  considerable  study,  based  partly  on 
experience  with  software  development  projects, 
the  author  concluded  that  a superior  reliability 
model,  incorporating  some  of  the  concepts  of 
the  foregoing  models,  could  be  constructed  if 
one  first  thought  in  terms  of  execution  time,  i.e., 
the  actual  processor  time  utilized  in  executing 
the  program,  and  then  determined  how  execu* 
tion  time  and  calendar  time  were  related.  Exe- 
cution time  appeared  to  be  the  best  practical 
measure  for  characterizing  the  stress  placed  on 
software.  Furthermore,  one  could  relate  the 
error  correction  rate  to  the  instantaneous 
failure  rate  during  test,  eliminating  the  need  for 
developing  an  error  correction  model.  Finally, 
execution  time  could  be  related  to  calendar 
time  in  a simple  and  straightforward  fashion 
because  the  relationship  between  the  two  is 
generally  paced  at  any  given  time  by  one  of  the 
following  limiting  resources:  failure 

identification  personnel,  failure  correction  per- 
sonnel, or  computer  time.  The  failure 
identification  personnel  are  usually  the 
members  of  a teat  team  and  the  failure  correc- 
tion personnel  are  usually  the  original 
designers. 

Included  in  the  theory  is  a sutistical  pro- 
cedure for  continually  re-estimating  the  param- 


eters of  the  reliability  model  during  a test  phase 
of  a software  project.  Using  the  time  intervals 
between  failures  experienced  during  this  phase, 
one  can  estimate: 

(a)  the  total  number  of  failures  possible  dur- 
ing the  maintained  life  of  the  software 
(and  the  associated  number  of  errors), 

(b)  the  initial  MTTF  before  debugging 
started, 

(c)  the  present  MTTF, 

(d)  the  additional  number  of  failures  (or 
erron)  that  must  be  detected  in  order  to 
reach  whatever  MTTF  objective  has  been 
established  for  the  project,  along  with 
the  additional  number  of  computer  hours 
of  testing  required,  and 

(e)  the  additional  calendar  time  in  days 
required  to  meet  the  MTTF  objective. 

All  of  these  estimated  quantities  are  generated 
in  terms  of  a maximum  likelihood  value  and 
confidence  intervals  (typically  50,  75,  90,  and 
.95  percent). 

rv.  EXECUTION  TIME  COMPONENT 

OF  MODEL 

The  execution  time  model  is  based  on 
certain  assumptions  that  appear  to  be  met  rea- 
sonably well  by  most  execuuble  programs  and 
most  development  projects,  assuming  a prop- 
erly planned  and  executed  test  effort: 

Assumptton  1:  Tests  are  representative  of 
the  environment  in  which  the  program  will  be 
used  and  are  continuously  global. 

Assumption  2:  Failures  are  independent  of 
each  other  and  distributed  at  any  time  with  a 
constant  average  execution  time  occurrence  rate 
that  is  proportional  to  the  number  of  errors 
remaining. 

Assumption  3:  All  failures  are  observed. 

Assumption  4:  The  error  correction  rate  is 
proportional  to  the  failure  detection  rate  at  all 
times. 

The  discussion  of  these  assumptions  that  fol- 
lows is  supplemented  by  details  in  Appendix  B. 

It  is  not  necessary  that  testing  exhaust  all 
possible  environmental  or  input  conditions: 
however,  test  input  sets  should  be  selected  ran- 
domly from  the  complete  set  of  input  sets.  By 
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“continuously  global”  it  is  meant  that  the 
group  of  tests  ttut  occupy  a small  testing  inter* 
val  (perhaps  a calendar  day  or  so)  exercise 
essentially  all  the  code  of  the  program.  If  tests 
are  selected  so  that  only  module  A is  tested  on 
day  1 and  module  B is  tested  on  day  2,  esti- 
mates made  on  either  day  will  not  be  represen* 
utive  of  the  program.  If  the  tests  cannot  be 
“continuously  global”  because  testing  starts 
before  integration  is  complete  or  because  one 
component  (e.g.,  the  operating  system)  Is 
thoroughly  tested  before  the  others  are,  the 
situation  is  analogous  to  the  addition  of  design 
changes.  See  Appendix  C for  methods  of  han- 
dling this  condition. 

Note  that  the  constant  execution  time 
occurrence  rate  postulated  in  Assumption  2 
implies  that  failures  constitute  a piecewise  Pois- 
son process  (the  parameter  changes  whenever 
an  error  is  corrected)  with  execution  time 
between  failures  distributed  piecewise  exponen- 
tially. Studies  in  progress  using  real  project 
dau  appear  to  be  supporting  this  assumption,  to 
a first  approximation. 

Assumption  3 is  necessary  because  a 
failure  can  sometimes  be  subtle.  Subtle  failures 
can  occur  for  hardware  also,  but  probably  less 
frequently  than  for  software.  If  some  failures 
are  missed,  estimation  of  the  program’s  reliabil- 
ity will  be  optimistic.  The  author  found  that 
one  is  more  likely  to  miss  failures  in  the  opera- 
tional period  of  the  program  than  the  test 
period  since  heightened  attention  to  failures  is 
characteristic  of  most  test  periods.  Hence,  the 
estimate  of  the  program's  reliability  is  still  rela- 
tively pessimistic,  which  is  satisfactory  from  a 
practical  viewpoint. 

The  determination  of  which  departures  of 
operation  from  program  requirements  are 
"unaccepuble”  and  therefore  failures  is  up  to 
the  user  or  system  engineer,  as  long  as  the 
determination  is  made  consistently  throughout 
a projea.  For  example,  one  nuy  only  consider 
program  crashes  requiring  interruption  of  pro- 
cessing and  program  reload  as  unaccepuble. 
On  the  other  hand,  any  situation  in  which  sys- 
tem requirements  are  not  met,  such  as  an 
inconea  title  on  a report,  may  be  considered  as 
a failure.  One  may  even  wish  to  classify 
failures  by  severity  and  obtain  reliability  meas- 
ures for  each  class. 


In  Assumption  4,  the  consunt  of  propor- 
tionality between  the  error  correction  rate  and 
the  failure  detection  rate  is  called  the  error 
reduction  factor  B . It  can  account  for: 

(a)  error  growth  due  to  new  errors  that  are 
spawned  in  the  process  of  correcting 
errors  or  that  emanate  from  the  added  or 
modified  code  resulting  from  design 
changes, 

(b)  errors  found  by  reading  that  was  stimu- 
lated by  the  detection  of  a related  error 
during  test,  and 

(c)  a proportion  of  failures  whose  causative 
errors  cannot  be  determined  and  hence 
cannot  be  corrected. 

The  latter  situation  may  exist  because 
insufficient  dau  have  been  gathered  to  track 
the  causes  of  the  failures  and  the  failures  can- 
not be  reproduced.  This  sute  of  affairs  often 
occurs  for  operating  systems  in  compuution 
centers.  -See  Appendix  D for  more  details. 

Based  on  the  assumptions  that  have  been 
made,  it  has  been  shown  [12]  that  the  net 
number  of  errors  detecud  and  corrected  n is  an 
exponential  function  of  the  execution  time  r: 


n 


1 — exp 


Ct 

MoTo 


(1) 


where  iVo  is  the  number  of  inherent  (unre- 
duced by  debugging)  errors.  Mg  is  the  toul 
number  of  failures  possible  during  the  main- 
tained life  of  the  software,  Tg  is  the  MTTF  at 
the  start  of  test,  and  C is  the  testing  compres- 
sion factor.  “Maintained  life”  refers  to  the 
period  extending  from  the  start  of  test  to  the 
discontinuance  of  program  failure  correction. 
Nou  that  Mg  is  also  the  number  of  failures  that 
must  be  experienced  to  remove  all  errors.  The 
testing  compression  factor  indicates  the  extent 
to  which  operational  runs  occurring  under 
duplicate  conditions  have  been  reduced  or  elim- 
inated during  test  by  test  selection  and  design. 
For  example,  one  hour  of  test  may  represent  12 
hours  of  actual  operation.  Thus  C represents 
the  ratio  of  equivalent  operation  time  to  test 
time.  A more  detailed  ti^ussion  is  given  in 
(121. 
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Now  it  will  be  seen  that  failures  and 
errors  are  related  through 

1^0  - BMo  (2) 


and 


n ^ Bm  . 


(3) 


where  m is  the  number  of  failures  experienced. 
Expressions  in  terms  of  failures  are  most  use* 
fuL  since  they  are  most  readily  measurable. 
Hence  from  (1),  (2),  and  (3)  we  obtain 


Cr 

m • Mo 

1 - exp 

“ MoTo] 

(4) 


The  present  MTTF  fis  then  shown  to  be 


T 


Tq  exp 


Cr 
Mo  To 


(5) 


MTTFs  are  measured  in  execution  time.  Note 
that  the  present  MTTF  increases  as  testing 
proceeds.  Reliability  R for  an  operational 
period  r'  is  given  by 


R — exp 


(6) 


By  eliminating  r between  (4)  and  (5), 
solving  for  m,  evaluating  the  resultant  equation 
at  /R|.  Ti  and  mi,  Ti.  and  subtracting,  we 
obtain  an  equation  for  the  number  of  failures 
A/n  that  must  be  experienced  to  increase  MTTF 
from  fi  to  Til 


A/rr 


Mo  To 


(7) 


Data  taken  on  six  different  development 
projects  indicates  that  the  model,  characterized 
by  (4),  is  followed  quite  well.  An  example  is 
shown  in  Fig.  1.  where  number  of  failures  is 
plotted  against  execution  time.  The  vertical 
axis  has  been  transformed  so  that  dau  that  fit 
(4)  plot  as  a straight  line  (statistical  variation 
around  the  line  is  expected  and  must  be 
allowed  for).  It  will  be  observed  that  the  fit  is 
quite  good.  Similar  results  have  been  obtained 
on  other  projects.  It  should  be  noted  that 
changes  in  Mo  or  To  due  to  substantial  design 
changes  or  substantial  amounts  of  cpde  added 
will  result  in  a change  in  the  slope  of  the  plot 
when  they  occur. 
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Fig.  1.  Veriflcation  of  exccation 
time  model. 
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If  (S)  is  solved  for  r and  evaluated  at 
r|,  f)  and  rj,  Ti  and  the  resultant  equations 
are  subtracted,  we  obtain  the  relationship  that 
specifies  the  additional  execution  time  Ar 
required  to  increase  the  MTTF  from  Ti  to  Tj: 


Ar 


MqTq  T\ 

c “ r, 


(8) 


V.  CALENDAR  TIME  COMPONENT 
OF  MODEL 

In  most  projects  during  test,  one  is  fairly 
tightly  constrained  as  to  the  quantities  of  failure 
identification  (test  team)  personnel,  failure 
correction  (original  designer)  personnel,  and 
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computer  time  available.  Increases  in  any  of 
these  items  generally  require  a long  lead  time; 
in  fact,  as  Fred  Brooks  [2.  p.  25]  points  out. 
they  may  actually  be  counterproductive. 
Failure  identification  is  the  process  of  compar- 
ing test  results  against  program  requiremenu  to 
esublish  the  failures  that  have  occurred. 
Failure  correction  is  the  process  of  determining 
the  error  that  caused  a failure,  removing  the 
error,  and  proving  that  the  failure  no  longer 
occurs.  ‘‘Computer  time”  is  the  measure  that 
is  used  for  allocating  computer  resources;  it  is 
not  necessarily  the  same  as  execution  time. 
Frequently,  it  is  the  residence  time  of  the  pro- 
gram in  the  machine  (perhaps  divided  by  the 
number  of  programs  that  can  be  concurrently 
resident  in  the  case  of  multiprogramming). 

Failure  identification  personnel  and  failure 
correction  personnel  are  considered  as  separate 
resources.  Failure  identification  and  correction 
are  usually  performed  by  different  people  in  the 
case  where  this  model  will  be  most  frequently 
applied,  the  system  test  phase  of  a project  of 
reasonable  size.  See  Appendix  D for  a discus- 
sion of  bow  to  handle  the  case  where  these 
resources  are  merged. 

A test  effort  will  consist  of  from  one  to 
three  periods,  each  characterized  by  a different 
limiting  resource.  Typically,  one  identifies  a 
large  number  of  failures  separated  by  short 
time  intervals  at  the  start  of  test,  and  testing 
must  be  stopped  from  time  to  time  in  order  to 
let  the  people  who  are  fixing  the  errors  keep  up 
with  the  load.  As  testing  progresses,  the  inter- 
vals between  failures  become  longer  and  longer 
and  the  debuggers  are  no  longer  fully  loaded, 
but  the  test  team  becomes  the  bottleneck. 
Finally,  at  even  longer  failure  intervals,  the 
capacity  of  the  computing  facilities  is  limiting. 
T^  debugging  process  model  that  is  incor- 
porated in  this  reliability  theory  utilizes  the 
knowledge  of  these  pacing  resources  to  relate 
execution  time  on  the  computer  with  the  pas- 
sage of  calendar  time. 

This  model  makes  the  following  assump- 
tions: 

Assumption  S:  The  quantities  of  the 
resources  failure  identifiation  personnel, 
failure  correction  personnel,  and  computer  time 
that  are  avaiiabk  are  constant  for  the  remainder 
of  the  test  period. 


Assumption  6:  Resource  expenditures  x 
may  be  approximated  by 

X - Obr  + fi^m  , (9) 

where  Ar  is  the  increment  of  execution  time. 
Am  is  the  increment  of  failures  experienced.  0 
is  the  partial  resource  expenditure  rate  with 
execution  time,  and  m is  the  partial  resource 
expenditure  rate  per  failure. 

Assumption  7:  Failure  identification  per- 
sonnel can  be  fully  utilized,  computer  utiliza- 
tion is  constant,  and  failure  correction  person- 
nel utilization  is  esublished  by  limiution  of 
error  queue  length  for  any  debugger.  Error 
queue  length  is  determined  by  assuming  that 
failure  correction  is  a Poisson  process  and  that 
servers  are  randomly  assigned  in  time. 

It  should  be  noted  that  the  availability  of  a 
fixed  quantity  of  resources  does  not  insure  that 
they  are  all  used.  In  fact,  at  any  given  time, 
only  the  limiting  resource,  at  most,  will  be  fully 
utilized,  and  the  other  resources  will  not. 

In  justification  of  Assumption  6,  note  that 
identification  of  failures  requires  work  and 
computer  time  that  increases  with  the  amount 
of  execution  time  used  and  the  related  amount 
of  test  output  generated.  Also,  each  failure 
requires  work  for  verification  and  determination 
of  its  specific  nature.  Correaion  of  failures 
requires  work  and  computer  time  that  is  pro- 
portional to  the  number  of  failures.  Computer 
time  is  expended  in  both  identification  and 
correction  of  failures;  hence,  the  time  used  may 
be  expected  to  depend  on  both  amount  of  exe- 
cution time  and  number  of  failures. 

Assumption  7 implies  that  the  time 
required  to  correct  a failure  is  independent  of 
the  time  required  to  correct  other  failures  and 
the  probability  that  an  error  is  fixed  during  any 
infinitesimal  time  period  of  debugging  is  equal 
to  the  probability  that  it  is  fixed  during  any 
other  such  period.  Experience  has  demon- 
strated that  this  is  a fairly  good  represenution, 
although  it  may  somewhat  overestimate  the 
proportion  of  errors  with  short  fix  times. 

The  ‘‘continuously  global  testing” 
assumption  made  for  the  execution  time  com- 
ponent of  the  software  reliability  model  implies 
that  the  assignment  of  failures  to  failure  conec- 
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tion  personnel  will  be  essentially  random  in 
time,  since  no  section  of  code  (and  the  pro- 
grammer responsible  for  it)  is  being  favored  in 
the  testing  process.  Thus  each  debugger  may 
be  viewed  as  a Poisson  server  in  a queueing 
system  with  an  input  of  identified  failures. 
Experience  has  indicated  that  testing  is  stopped 
whenever  an  excessive  backlog*  of  failures  is 
building  up  for  any  one  debugger  and  hence 
impeding  the  identification  and  correction  of 
other  failures  in  his  program  area.  The  value 
of  the  failure  correction  personnel  utilization 
factor  pr  ntay  therefore  be  determined,  by  lim- 
iting the  queue  length  (at  a given  probabil- 
ity level  P,-).  as 


P k Pk  “ PkPk^k 


that  lie  within  that  range,  where  {k,  k')  have 
the  values  (C  F).  (F.  /).  and  (/,  C). 

Some  further  details  are  discussed  in 
Appendix  B. 


VI.  estimation  of  MODEL 

parameters 


Pf  ■ 


(10) 


Using  the  assumptions  that  have  been 
made,  calendar  time  intervals  can  be  related  to 
execution  time  intervals,  and  employing  (5),  to 
MTTFs.  [12]  The  calendar  time  interval  It  is 
given  by 


1 1 

n,  " TVj 

T — In 

r*. 

(11) 

The  index  k can  have  the  values  C.  F, 
or  I.  corresponding  to  the  computer-limited, 
failure-correction-personnel-limited,  or  failure- 
identificadon-personnel-limited  periods,  respec- 
tively. The  quantities  7,,^  and  represent  the 
MTTFs  at  the  boundaries  of  these  periods.  P is 
the  resource  quantity,  and  p is  the  resource 
utilization.  Ordinarily,  - 0 and  Pr  1 . 

The  boundaries  of  the  different  resource- 
Umited  periods  are  the  present  and  objective 
MTTFs  and  the  transition  points 


*In  nnwuntia  bacKloi  wt  do  not  count  failurts  whoM 
comcuoa  a dcftnad  Tor  rtaions  oilMr  thwi  avtiinSili- 
cy  of  ptnotuMl  to  itwm. 


A number  of  parameters  have  been 
defined  in  preceding  sections  that  must  be 
evaluated  in  order  to  specify  the  reliability 
model  completely.  The  essential  ones  are  listed 
in  Table  I.  The  weighted  average  of  parameter 
values  used  for  four  projects  for  which  com- 
plete dau  is  presently  available  is  given  where 
they  have  any  possible  significance  for  other 
projects.  The  parameters  are  grouped  into  four 
categories:  planned,  debug  environment,  test 
environment,  and  program.  Predictions 
affected  by  the  parameter  are  indicated. 

The  planned  parameters  Pc.  Pf  , and  Pi 
are  determined  by  the  computer  and  personnel 
resources  available.  The  parameter  Pc 
represents  computer  time  in  terms  of 
prescribed  work  periods.  For  example,  if  avail- 
able computer  time  per  week  is  75  hours  and 
the  prescribed  work  week  is  37. S hours,  than 
Pc~l. 

The  computer  utilization  factor  pc  should 
be  set  at  its  actual  value.  Note  that  this  factor 
is  often  either  implicitly  or  explicitly  controlled 
in  order  to  maintain  satisfactory  turnaround 
time.  The  failure  correction  personnel  utiliza- 
tion factor  Pf  is  computed  using  (10).  Exoeri- 
ence  thus  far  seems  to  indicate  that  controlling 
the  failure  queue  length  so  that  it  does  not 
equal  or  exceed  3 with  probability  0.9  appears 
reasonable;  these  values  nuy  be  refined  with 
more  knowteoge  of  the  effect  of  backed-up 
failures  on  the  debugging  process. 

The  Ml'  IF  objective  Tf  is  generally  set 
based  on  external  considerations  such  as 
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TABLE  I 

essential  parameters  of  reliability  model 


PanaiMtr 

DtSaitloa 

Cattfory 

av«r*g*  Valat 

Ovvr 

Projects  1>4 

PrceictioBS 

AffoctoS 

Pc 

available  compuur  time  (measured 
in  terms  of  prescribed  work 
periods) 

planned 

- 

Ar 

Pf 

number  of  available  failure 
correction  personnel 

planned 

- 

Ar 

P, 

number  of  available  failure 
identification  personnel 

planned 

- 

A/ 

Pc 

computer  utilization  factor 

planned 

Ar 

Pf 

failure  correction  personnel 
utilization  factor 

planned 

- 

Ar 

Tf 

objective  MTTF 

planned 

- 

A/n,  Ar, 
Ar 

average  computer  time  expended 
per  unit  execution  time 

debug 

environment 

1.68 

Ar 

•; 

average  failure  identification  work 
expended  per  unit  execution  time 

debug 

environment 

2.86 

Ar 

average  computer  time  rettoired 
per  failure 

debug 

•nvironment 

2.03  hr 

Ar 

Mf 

average  failure  correction  work 
required  per  failure 

debug 

environment 

6.42  hr 

Ar 

Pr 

average  failure  identification  work 
required  per  failure 

debug 

environment 

2.01  hr 

Ar 

C 

testing  compression  factor 

test 

environment 

12.5 

T,  Am, 
Ar,  Ar 

Mq 

number  of  failures  required  to 
expose  and  remove  all  errors 

proftram 

- 

r.  Am, 
Ar,  Ar 

n 

initial  MTTF  at  start  of  testing 

itrogram 

- 

T.  Am. 
Ar,  Ar 

economics  and  Uie  MTTFs  of  other  system 
components. 

The  debug  environment  parameters 
should  be  obtained  from  dau  uken  in  the 
actual  project  debug  environment  or  one  that 
closely  resembles  iu  By  “environment”  we  are 
referring  to  factors  such  as  batch  debugging  vs. 


interactive  debugging,  debugging  aids  available, 
computer  used,  language  used,  administrative 
and  documenution  overhead  associated  with 
corrections,  etc.  The  parameters  may  be 
evaluated  by  collecting  dau  on  failure 
identification  and  correction  work  and  computer 
usage  profiles  (with  execution  time)  and  fitting 
the  dau  with 
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X ™ tfr  + ijlMo 
|l  -expl- 


(13) 


MoTo 


using  a weighted  least  squares  criterion. 

It  appears  likely  that  some  or  ail  of  the 
debug  environment  parameters  will  prove  to  be 
relatively  constant  over  wide  varieties  of  pro* 
jects  or  classes  of  projects.  As  more  dau  is  col* 
lected,  it  should  be  possible  to  gain  a deeper 
understanding  of  what  ranges  of  values  these 
parameters  take  on  and  what  factors  they  are 
dependent  on.  This  knowledge  should  make  it 
possible  to  increase  the  accuracy  with  which  the 
calendar  time  required  to  meet  a mean  time  to 
failure  objective  can  be  predicted. 

The  testing  compression  factor  C depends 
on  the  test  environment  and  its  relationship  to 
the  actual  operating  environment  of  the  pro* 
gram.  More  specifically,  its  value  is  related  to 
the  extent  to  which  redundancy  due  to  runs 
being  made  under  identical  conditions  during 
the  operation  of  the  program  is  removed  during 
/ the  test  phase.  There  is  some  indication  that  C 
is  reasonably  suble  across  similar  test  environ* 
ments,  but  dau  from  more  projects  is  needed 
to  verify  this.  The  quantity  C nuy  be  com* 
puted  for  a project  after  a sufficient  period  of 
operation  has  occuned  following  the  test  phase 
so  that  operational  phase  initial  MTTF  To  can 
be  accurately  estimated.  By  use  of  (5)  and  the 
fact  that  the  MTTF  at  the  end  of  test  will  be 
equal  to  to,  we  obtain 


C 


iV/qTo 

r 


(14) 


where  To  is  the  MTTF  at  the  start  of  the  test 
phase  and  r is  the  total  execution  time  for  the 
test  phase.  If  there  is  no  good  way  of  estimat* 
ing  C for  a particular  project,  it  is  probably  best 
to  be  conservative  and  take  C - I . 

The  two  program  parameters  must  be 
initially  estimated  from  characteristics  of  the 
program  itself.  However,  once  dau  is  available 
on  intervals  between  failures  during  test  or 
operation,  these  parameten  may  be  rees* 
timated.  The  accuracy  with  which  they  are 


known  generally  increases  with  the  size  of  the 
sample  of  failures. 

The  parameter  <V/q  must  initially  be 
estimated  from  the  number  of  inherent  failures 
/Vq  and  the  error  reduction  factor  B , using  (2). 
Dau  can  be  developed  on  average  error  rates 
(erron  per  instruction)  at  the  surt  of  various 
phases  of  testing;  these  dau  will  permit  .Vg  to 
be  estimated.  Dau  taken  in  this  study  and  dau 
taken  by  Akiyama  (14)  and  Endres  [15]  give  a 
range  of  3.36  to  7.98  errors  per  thousand 
source  instructions  for  assembly  language  pro* 
grams  at  the  start  of  system  test;  the  weighted 
(by  number  of  instructions)  mean  is  S.43 
errors/K  inst.  For  a detailed  discussion  of  the 
determination  of  B.  see  Appendix  D. 

The  parameter  To  must  initially  be 
estimated  from 


!l 


^O- 




/KNo  ■ 


(15> 


where  / is  the  linear  execution  frequency  of  the 
program  or  the  average  instruction  execution 
rau  divided  by  the  number  of  instructions  in 
the  program  and  AT  is  an  error  exposure  ratio 
which  relates  error  exposure  frequency  to 
“error  velocity.”  The  error  velocity  is  the  rate 
at  which  errors  in  the  program  would  pass  by  if 
Che  program  were  executed  linearly.  The  error 
exposure  ratio  accounts  for; 

(a)  Code  is  not  “straight  line”  but  has  many 
loops  and  branches,  except  in  very  trivial 
cases,  and 

(b)  The  machine  sute  varies,  and  hence  the 
error  associated  with  an  instruction  may 
or  may  not  be  exposed  at  one  particular 
execution  of  the  instruction. 

At  present,  'K  must  be  determined  from  a 
similar  program.  It  may  be  possible  in  the 
future  to  relate  K to  program  structure  in  some 
way.  On  six  projects  for  which  dau  is  available, 
K ranges  from  1.54x10“’  to  2.99x10“’  . 
Therefore,  intervals  between  failures  typically 
represent  cycling  through  the  program  a large 
number  of  times. 

Some  of  the  practical  aspects  of  parameter 
estimation  are  discussed  in  Appendix  D. 
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VII.  PROGRAM  PARAMETER 

REESTIMATION 

Maximum  likelihood  estimation  is  used  to 
reestimate  A/q  and  Tq  as  testing  progresses. 
The  important  relationships  are  given  in 
Appendix  E.  A portable  FORTRAN  program 
has  been  developed  116], [17]  to  perform  the 
calculations,  so  a graphical  view  will  be  pro* 
vided  at  this  point. 

The  essential  data  required  for  the  reesti* 
mation  are  the  execution  time  intervals 
between  failures  experienced  in  testing.  A set 
of  such  intervals  have  been  plotted  against  the 
sequential  numbers  of  failures  in  Fig.  2.  The 
interval  plotted  for  failure  m represents  the  ' 
execution  time  between  failure  m-1  and  failure 
m.  The  set  of  intervals  constitutes  a sample 
from  a multivariate  probability  distribution. 

The  execution  time  model  provides  the 
basis  for  plotting  a curve  of  anticipated  mean 
failure  interval  vs.  failure  number.  This  curve 
is  determined  by  two  parameters,  the  initial 
MTTF  To  and  the  toul  expected  failures  Mo- 
Values  of  the  parameters  are  chosen  to  maxim- 
ize the  likelihood  of  occurrence  of  the  set  of 
intervals  that  have  been  experienced.  In  a very 
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Fig.  2.  Pregnai  parameter  reestimation 
and  project  statna  monitoring. 


approximate  sense,  the  curve  is  being  fit  to  the 
dau  indicated. 

It  will  be  seen  from  the  nature  of  the 
curve  that  To  can  be  estimated  more  accurately 
than  Mo  ■ The  parameter  To  is  the  vertical  axis 
intercept  of  the  curve  but  Mo  + \ is  the  hor* 
izonul  axis  value  that  the  curve  approaches 
asymptotically.  Note  that  the  number  of 
failures  to  date  can  be  read  from  the  figure. 
The  execution  time  that  has  elapsed  to  date  is 
the  sum  of  the  failure  intervals  experienced;  it 
is  approximated  by  the  area  under  the  curve. 

Since  reestimation  of  the  parameters  is  a 
sutistical  process,  one  also  wishes  to  determine 
confidence  intervals.  The  curve  should  now  be 
viewed  as  a band,  with  iu  thickness  increasing 
with  the  magnitude  of  the  confidence  interval 
(a  90  percent  confidence  interval  requires  a 
thicker  band  than  a 7S  percent  one).  The 
confidence  interval  for  To  is  given  by  the  range 
of  vertical  axis  intercepts.  The  confidence 
interval  for  Mo  is  given  by  the  range  of  asymp- 
totes approached. 

VUI.  MONITORING  TESTING 

PROGRESS 

Since  Mo  and  To  are  continually  refined 
by  reestimation  during  test,  it  is  easy  to  con- 
tinually reestimate; 

(a)  Present  MTTF  Tp.  using  (S)  with  t set 
equal  to  execution  time  used  to  date. 

(b)  Remaining  execution  time  Ar  required 
for  the  maximum  likelihood  estimate  of 
T to  reach  an  esublished  MTTF  objec- 
tive Tf,  using  (8)  with  T:  « fp  and 
r,  - Tp . 

(c)  Remaining  calendar  time  It  required  to 
reach  fp,  evaluating  (11)  over  the  inter- 
val (Tp.  Tp]. 

(d)  Remaining  failures  Am  required  to  be 
experienced  to  reach  Tp,  using  (7)  with 
Tj  * Tp  and  Tj  * Tp  . 

Some  of  these  quantities  may  be  viewed 
on  Fig.  2.  The  present  MTTF  is  esublished  as 
the  vertical  axis  value  of  the  intersection  of  the 
curve  with  the  vertical  line  that  represenu 
number  of  failures  to  date.  The  intersection  of 
the  MTTF  objective  line  with  the  curve  deter- 
mines the  number  of  additional  failures 
required  to  meet  the  objective  (increment 
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measured  along  horizontal  axis)  and  the  addi- 
tional execution  time  required  (area  increment 
under  the  curve). 

Confidence  limits  may  be  established  (in 
addition  to  the  "most  likely”  values)  by  carry- 
ing out  the  compuutions  for  corresponding 
confidence  limits  of  .V/q  and  fj  (e.g.,  the 
confidence  limits  for  Ar  are  the  remaining  exe- 
cution times  required  for  the  confidence 
bounds  of  Tto  reach  Tf).  The  confidence  lim- 
its may  be  seen  pictohally  on  Rg.  2 by  viewing 
the  curve  as  a band. 

The  total  set  of  results  is  useful  to 
managers  for  planning  schedules,  estimating 
sutus  and  progress,  and  determining  when  to 
terminate  testing.  The  7S  percent  confidence 
level  esumate  was  found  from  experience  to  be 
the  most  useful  from  the  manager's  viewpoint, 
although  the  50  percent.  90  percent,  and  95 
percent  confidence  levels  contribute  some 
added  insight. 

The  FORTRAN  program  oreviously  men- 
tioned also  computes  estimates  of  T/,,  Sm,  Ir. 
and  Ar  from  the  set  of  intervals  between 
failures  and  certain  required  model  parameters. 
A sample  report  output  by  this  program  for  an 
actual  project  is  shown  in  Rg.  3.  Note  that 
"999999”  indicates  “no  upper  limit." 

It  is  thus  possible  for  the  manager  to 
obtain  on  a regular  and  continual  basis,  during 
the  system  test  phase  of  a project,  information 
such  as  the  following  (75  percent  confidence 
intervals  will  be  suted): 

(a)  At  present  the  best  estimate  of  total  pos- 
sible failures  during  the  maintained  life 
of  the  program  is  142  with  a confidence 
interval  of  136  to  152. 

(b)  The  best  estimate  of  MTTF  prior  to  test- 
ing is  0.847  houn  (confidence  interval 
from  0.701  to  0.992  hours). 

(c)  The  present  MTTF  is  20.4  hours 
(confidence  interval  more  than  12.5 
hours).  This  represents  73.4  percent 
(confidence  interval  from  45.1  percent  to 
100  percent)  of  the  MTTF  objective  of 
27.3  hours. 

(d)  In  order  to  meet  this  objective,  the  best 
estimate  of  the  additional  number  of 
failures  to  be  experienced  is  2 
(confidence  interval  from  0 to  7). 


(e)  It  is  estimated  (hat  meeting  the  objective 
will  require  2.46  additional  computer 
hours  of  testing  (confidence  interval 
from  0 to  7 94)  and  one  wot  king  day 
(confidence  interval  from  0 to  4). 

(f)  Expected  completion  date  is  11/12/73 
(confidence  interval  from  11/9/73  to 
11/16/73). 

It  should  be  noted  that  the  confidence  intervals 
are  of  greatest  significance  to  the  manager, 
although  the  maximum  likelihood  estimate  is  of 
interest. 

Although  the  estimation  program  has  pro- 
ven to  be  of  most  value  during  the  system  test 
phase,  nothing  prevents  it  from  being  applied 
to  individual  programmers  who  are  testing  sub- 
systems. If  the  subsystems  are  too  small,  how- 
ever, estimates  that  are  made  may  have  wide 
confidence  intervals  for  a large  part  of  the  test 
period,  and  hence  be  of  limited  value.  If  calen- 
dar time  predictions  are  to  be  accurate,  you 
may  wish  to  use  resource  parameter  values  that 
are  either  measured  or  adjusted  to  apply  to  the 
panicuiar  individuals  involved. 

The  program  can  also  be  used  during  the 
operational  phase  of  a project.  If  the  software 
is  being  mainuined  (i.e.,  failures  are  being 
corrected  provided  the  errors  causing  them  can 
be  found),  the  operational  phase  is  handled  just 
like  a test  phase  except  the  testing  compression 
factor  C is  set  to  1.  If  failure  intervals  from  a 
test  phase  and  the  operational  phase  are  used 
together,  the  former  intervals  should  be 
increased  by  a factor  of  C (and  the  parameter  C 
should  be  set  to  I as  before).  If  the  software  is 
no  longer  being  maintained,  software  reliability 
estimation  programs  (161, [17]  can  still  be 
employed.  MTTFs  with  confidence  intervals 
can  be  estimated,  but  A/q  will  generally  be 
infinite  (.B  is  0.  allowing  /Vq  to  be  finite)  and 
the  quantities  Am,  Ar,  and  Ar  required  to  reach 
the  MTTF  objective  cannot  be  determined. 

Note  that  the  program  correctly  estimates 
the  situation  with  regard  to  failures  at  all  times; 
determination  of  inherent  errors  Yq  or  addi- 
tional net  errors  A/i  that  must  be  corrected  to 
atuin  the  MTTF  objective  requires  knowledge 
of  the  error  reduction  factor  B . 


I 
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SOFTWARE  RELIABILITY  PREDICTION 
A ^ROJECT  1 


BASED  ON  SAMPLE  OP  136  TEST  FAILURES 
EXECUTION  TINE  IS  25.34  HRS 

NTTF  OBJECTIVE  IS  27.80  HOURS 

CALENDAR  TIME  TO  DATE  IS  96  DAYS 
PRESENT  DATE: 11/  9/73 


951 

CONP. 

90% 

LIMITS 

75% 

<^0% 

MOST 

LIKELY 

50% 

CONF. 

75% 

LIMITS 

90% 

95% 

TOTAL  FAILURES 

136 

136 

148 

152 

163 

182 

INITIAL  NTTF (HR) 

0.522 

0.617 

0.701 

0.744 

20.4 

0.949 

0.992 

1.08 

1.17 

PRESENT  NTTF (HR) 

999999 

999999 

999999 

30.9 

14.5 

12.5 

9.53 

7.05 

PERCENT  OF  OBJ 

100.0 

100.0 

100^ 

100.0 

73.4 

52.0 

45.1 

34.3 

25.4 

• •• 

ADDITIONAL  REQUIREMENTS  TO  MEET  HTTP 

OBJECTIVE  ••• 

FAILURES 

0 

0 

0 

0 

2 

5 

7 

12 

23 

EXEC.  TINE (HR) 

0 

0 

0 

0 

2.46 

6.09 

7.94 

12.4 

19.4 

CAL.  TINE (DAYS) 

0 

0 

0 

0.958 

2.85 

4.03 

7.39 

13.8 

COMPLETION  DATE  110973  110973  110973  110973  111273  111473  111673  112173  112973 


Fit*  3.  Sample  project  status  report  as  generated  by  FORTRAN  protram. 


IX.  APPLICATION  OF  THEORY  TO 

PROJECT  SCHEDULING  AND 

MONITORING 

The  software  reliability  theory  described 
in  this  paper  is  currently  being  applied  to  the 
monitoring  of  a number  of  software  develop- 
ment projects,  as  indicated  in  Table  II.  The 
projects  are  being  conducted  both  inside  and 
outside  Bell  Laboratories.  They  vary  in  size 
from  3 to  about  300  people  and  about  5000  to 
over  2,000,000  instructions  and  include  real- 
time control  systems,  analysis  tools,  interactive 
systems,  military  systems,  business  systems, 
and  systems  with  large  dau  bases. 

The  MTTFs  prediaed  at  the  end  of  sys- 
tem test  for  the  four  projects  that  have  com- 
pleted their  life  cycles  are  compared  with  meas- 
ured MTTFs  in  Table  HI.  In  all  four  cases,  the 
measured  value  lies  within  the  SO  percent 
confidence  range  of  the  prediction. 

Two  typical  plots  from  Project  1 are 
shown  in  Figs.  4 and  S.  The  center  curve  is  the 
maximum  likelihood  estimate  and  the  outer 


curves  represent  the  bounds  of  the  75  percent 
confidence  interval,  in  each  case.  Both  figures 
represent  running  histories  of  predictions 
throughout  the  course  of  the  project,  as  indi- 
cated by  the  dates  on  the  horizonul  axis.  Fig. 
4 is  a plot  of  present  MTTF;  it  is  useful  for 
indicating  progress  toward  the  MTTF  objective 
(some  managers  may  prefer  the  alternative  plot 
of  “percent  of  objective  attained”).  The  upper 
confidence  limit  for  this  quantity  can  be  noisy 
when  only  a few  errors  remain,  since  a slight 
shift  in  the  number  remaining  can  affect 
present  MTTF  drastically.  It  is  very  common 
for  the  upper  confidence  limit  for  present 
MTTF  to  cease  to  exist  at  the  end  of  the  test 
period  of  a project  with  very  few  errors  remain- 
ing. Fig.  5 is  a plot  of  the  current  estimate  of 
project  completion  date.  Note  that  for  a good 
part  of  Oaober,  the  75  percent  confidence 
interval  does  not  enclose  the  completion  date 
predicted  at  the  end  of  system  test.  This  is  to 
be  expected;  about  1/4  of  the  time  it  will  not. 
The  95  percent  confidence  interval  does  enclose 
the  final  predicted  completion  date  almost  all 
the  time. 
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TABLE  II 

APPLICATION  OF  SOFTWARE 
RELIABILITY  THEORY  TO 
SOFTWARE  DEVELOPMENT 


PROJECTS 

Nambtr  o( 

Suns  Pnjtca 

Complete  data  taken— system 

test  and  operational  phases  4 

Data  available  from  system 
test  phase,  currently  talcing 
dau  on  operational  phase  1 

Complete  data  taken— system 

test  phase  3 

Currently  taking  data— system 

teat  phase  1 

Actively  preparing  to  take 

data— system  teat  phase  11 

Strong  interest  in  applying  6 

Analyzing  or  planning  to  ana- 
lyze dau  from  past  projects  3 

Total  29 


Some  projecu  pass  through  several  ver- 
sions with  subsuntial  design  changes  between 
them.  Figs.  6 and  7 were  taken  from  such  a 
project.  The  effects  of  the  changes  are  exag- 
gerated here  because  of  the  compression  of  the 
horizonul  axis  necessary  to  accommodate 
almost  1-1/2  years  of  data.  Fig.  6 is  a plot  of 
the  estimated  total  failures;  note  the  substantial 
increase  that  occurs  at  each  design  change  (the 
change  in  the  estimate  is  not  insuntaneous; 
there  is  a small  time  lag  until  the  estimation 
algorithm  realizes  that  the  change  has 
occurred).  Also  observe  the  sharp  increase  in 
size  of  the  confidence  interval  and  subsequent 
reduction  as  the  estimation  algorithm  copes 
with  the  new  uncertainty.  Fig.  7 is  a plot  of 
present  MTTF;  note  how  MTTF  drops  when 
substantial  design  changes  are  made  and  slowly 
rises  during  periods  of  steady  debugging. 
Further  discussion  of  the  handling  of  design 
changes  is  contained  in  Appendix  C. 
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Studies  are  currently  in  progress  to  apply 
the  reliability  monitoring  techniques  to  operat- 
ing system  software  in  computation  centers. 
Compuution  center  managers  commonly  have 
the  problem  of  reconciling  demands  for  new 
software  features  and  improvements  with 
requirements  for  reliability  of  service.  Adding 
new  features  usually  results  in  a period  of  relia- 
bility degradation,  until  the  new  features  are 
thoroughly  exercised  and  errors  corrected. 
Decisions  as  to  when  to  add  new  features  have 
previously  been  based  on  intuition.  The  new 
reliability  monitoring  techniques  (using  present 
MTTF)  should  make  it  possible  to  set  low  and 
high  service  objectives  and  control  addition  of 
new  features  to  keep  MTTF  approximately 
between  these  bounds  (new  features  would  be 
added  when  the  high  objective  is  exceeded  and 
a freeze  on  changes  would  be  imposed  when 
MTTF  drops  below  the  low  objective). 


X.  APPLICATION  OF  THEORY  TO 

SYSTEM  ENGINEERING 

• The  matmer  in  which  this  software  relia- 
bility theory  has  been  developed  results  in  a 
compatibility  with  hardware  reliability  theory 
that  permits  combination  of  hardware  and 
software  components  in  determining  overall 
reliability  of  complete  systems.  The  techniques 
of  reliability  budgeting  or  allocation  commonly 
used  in  system  engineering  can  therefore  be 
applied  to  systems  involving  hardware  and 
software  components. 

Two  approaches  to  this  combinatoric 
problem  are  presented.  The  first  deals  with 
components  that  function  concurrently;  it 
represents  the  conventional  combinatoric 
approach  used  for  hardware  systems.  The 
second  approach  relates  to  components  that 
function  sequentially.  It  was  developed  origi- 
nally for  pure  software  systems  and  is  probably 
most  applicable  to  them;  however,  it  can  also 
be  applied  to  hardware  or  mixed  systems  if  the 
hardware  components  satisfy  the  assumed  con- 
ditions. 

A system  is  considered  to  be  composed  of 
concurnmly-fitnctioning  components  if  the  satis- 
factory operation  of  the  system  is  dependent  on 
continuous  satisfactory  operation  of  each  com- 
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Flf.  4.  Present  MTTF  hbtory 
for  Project  1. 
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Fl|.  5.  Predicted  compietion  dote 
history  for  Project  1. 
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Fig.  6.  Predicted  totsl  fsiinres 
history  for  Project  5. 


Fig.  7.  Present  MTTF  history 
for  Project  S. 
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TABLE  III 

COMPARISON  OF  MEASURED  AND  PREDICTED  MTTF  (HOURS) 


PT«l«Ct 

PtaiMt 

PtoIwc 

1 

2 

3 

4 

Measured  (use  period) 

14.6 

31.4 

30.3 

9.2 

Predicted  (at  end  of  test) 

Max.  Likelihood  Value 

20.4 

43.5 

30.4 

14.5 

50%  Confidence  Range 

14.5-30.9 

>24.5 

>16.0 

7.6-27.9 

75%  Confidence  Range 

>12.5 

>19.8 

>11.9 

5.7-39.4 

90%  Confidence  Range 

>9.5 

>12.1 

>5.8 

>18 

95%  Confidence  Range 

>7.1 

>5.8 

>0.8 

>0.7 

pooent  or  set  of  components  (or  altemstively, 
if  failure  of  any  component  or  component  set  at 
any  dme  can  aflea  system  operation).  “Com- 
ponent set’*  is  mentioned  so  that  the  case  of 
multiple  redundant  components  is  included, 
wtare  the  entire  set  must  fail  before  system 
operatioit  is  affect^ 

The  foresoint  type  of  system  is  analyzed 
by  drswins  • **failute  logie’*  or  **succe^ul 
operation  logic’*  diagram  of  the  system  and 
applying  the  following  combinatoric  rules  to  the 
A^  or  OR  combinations  indicated: 

(a)  V att  k components  must  function  sue- 
cesaftiUy  for  system  success  (AND  con- 
dition), or  the  failure  of  any  of  the  ic 
components  will  cause  system  failure, 
the  system  reliability  R is  i^ven  by 

R - n R,  . (16) 

1*4 

where  R,  are  the  component  reliabilities. 

(b)  If  successful  functioning  ^ any  at  k 
components  will  result  in  system  success 
(OR  condition),  or  the  failure  of  all  A: 
components  b required  for  system 
failure,  the  system  reliability  is  given  by 

R - 1 - n (1  - R.)  . (17) 
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This  formula  is  derived  from  the  princi- 
ple that  the*  overall  failure  probability  is 
equal  to  the  product  of  the  individual 
figure  probabilities. 

If  the  component  characteristics  are 
specified  in  terms  of  MTTFs,  reliabilities  are 
determined  (assuming  exponential  failure  dis- 
tributions) from  (6). 

The  fact  that  software  components  are 
involved  requires  that  some  special  factors  be 
considered.  One  must  be  certain  that  all  the 
“times'*  are  either  equivalent  or  reduced  to  an 
equivalent  base.  For  example,  hardware 
MTTFs  are  generally  suted  in  actual  calendar 
time,  while  software  MTTFs  are  usually  suted 
in  execution  (CPU)  time.  If  two  hours  of 
calendar  time  elapse  on  the  average  for  every 
hour  of  execution  time  (i.e.,  processor  utiliza- 
tion is  0.5),  the  two  different  time  measures 
must  be  referred  to  a common  base.  Nou  that 
if  the  system  being  considered  is  divided  into 
several  software  componenu,  the  adjustment 
facton  will  change,  since  the  utilization  factors 
of  each  of  the  componenu  must  be  used. 

On  the  other  hand,  software  MTTFs  are 
sometimes  stated  in  calendar  time.  Now,  no 
adjustmenu  are  required.  However,  the  MTTF 
of  a software  component  will  no  longer  be  con- 
stant when  the  component  is  moved  to  a 
different  sysum,  even  if  the  processor  speed  is 
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the  same.  The  MTTF  is  then  adjusted  by  mul- 
tipying  by  the  ratio  of  utilizations.  It  is  prob* 
ably  best  to  measure  component  MTTFs  in 
terms  of  execution  time.  They  can  then  be 
adjusted  to  calendar  time  to  fit  any  system  utili* 
zation. 

Consider  a simple  example.  Fig.  8 
represents  a time-sharing  system  running  on  a 
multiprocessor  machine.  The  time-sharing  sys- 
tem runs  under  a general-purpose  operating 
system.  The  reliabilities  of  the  hardware  for  a 
lO-hour  prime  shift  are  labeled.  The  operating 
and  time  sharing  systems  have  MTTFs  meas- 
ured in  txtcutioH  time  of  20  hours  and  40  hours, 
respectively.  Each  runs  about  20  percent  of  the 
time  and  the  other  60  percent  of  the  processor 
capacity  is  used  by  application  programs.  We 
wish  to  find  the  prime  shift  reliability  of  the 
entire  system. 

The  calendar  time  MTTFs  enll  be  100 
hours  and  200  hours,  respectively,  and  the 
prime  shift  reliabilities  are  found  using  (6) 


Fig.  S.  Failoie  logic  diagram. 


as  0.90S  and  0.9S1.  The  combined  reliability  of 
the  two  processors  is  found  from  (17)  as 
0.9999.  Combining  all  the  “series”  elements 
by  use  of  (16),  we  find  an  overall  system  prime 
shiA  reliability  of  0.S56. 

A system  is  considered  to  be  composed  of 
teguentiaUy-ftiHaionmg  components  if  only  one 
component  functions  at  a time  and  the  satisfac- 
tory operation  of  the  system  is  dependent  on 
the  sa^actory  functioning  of  each  component 
when  it  is  aaiee.  This  situation  is  pertiicularty 
characteristic  of  a system  composed  of  a 
number  of  modules,  e^  of  which  has  control 
at  a given  time.  Littlewood  (191  has  developed 
a model  for  such  systems.  Tlie  model  assumes 
that  the  system  consists  of  k components 


among  which  control  is  switched  randomly 
according  to  a semi-Markov  process;  i.e.,  the 
transition  probabilities  between  components  are 
dependent  only  on  the  identity  of  the  preceding 
component.  The  probability  distributions  of  the 
sojourn  times  of  the  components  in  the  active 
sute  are  not  restricted  in  any  way.  Failures  for 
the  i-th  component  are  assumed  to  occur  in 
accordance  with  a Poisson  process  with  rate  X,. 
If  the  X,  are  small,  then  tte  system  process  is 
Poisson  with  rate  X given  by 


2 ^ PiPiji^ijkt 

^ 

LI  PiPijit-.j 


The  Pi  are  the  equilibrium  probabilities  given 
by 

P/  “ £ PjPjt  • (19) 

The  transition  probability  from  component  i to 
component  j is  given  by  p,),  and  M'iy  is  the  mean 
duration  spent  in  component  / before  switching 
to  component^ 

The  rate  X,  is  the  reciprocal  of  the  MTTF 
T,  (12,  Eq.  (7)],  and  hence  it  is  readily  derived 
from  reliabiUty  by  the  use  of  (6). 

It  should  be  noted  that  combination  of 
large  numbers  of  small  modules  is  generally  not 
practical  if  knowledge  of  T for  the  modules 
must  be  gained  by  estimation,  due  to  the  prob- 
lems of  limited  failure  sample  size. 


XI.  CONCLUSION 

The  theory  outlined  in  this  paper  has 
proved  to  be  a good  framework  for  understand- 
ing, measuring,  and  predicting  the  failure  pro- 
cess in  computer  programs  in  terms  of  reliabil- 
ity and  mean  time  to  failure.  Furthermore,  it 
constitutes  an  approach  that  is  compatible  with 
hardware  reliability  theory  u to  combination  of 
components  and  thus  permits  reliability  analysis . 
of  hardware-software  systems.  The  tl^ry  has 
been  applied  to  sever^  software  development 
prqjecu  of  different  kinds  and  subnantial 
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experience  has  been  gained  in  its  use.  It  is 
currently  being  employed  or  about  to  be 
employed  on  a larger  set  of  projects.  Studies 
now  in  progress  indicate  that  the  theory  also 
holds  considerable  promise  of  usefulness  in 
monitoring  compuution  center  service  reliabiU 
ity  and  employing  this  Itnowledge  in  controlling 
the  timing  of  software  changes. 


APPENDIX  A 
GLOSSARY 


eompuur  timt;  the  measure  used  for  allocating 
computer  resources.  It  is  often  the 
reskieiice  time  of  the  program  in  the 
twetiifia.  (perhaps  divided  by  the  number 
of  programs  that  can  be  concurrently 
resident  in  the  case  of  multiprogramming) . 

amUmnafy  gMttt  a property  of  a test  effort;  a 
test  effort  is  contiiuioualy  global  if  essen- 
tially all  code  is  exercised  in  each  of  a set 
of  test  intervals,  with  lengths  of  perhaps  a 
day  apiece; 

error:  the  software  defect  that  causes  a failure. 

execution  time:  operation  time  of  the  central 
processor  (s). 

failure:  an  unacceptable  departure  of  program 
operadon  from  program  requirements. 

fiubtre  correction:  the  process  of  determining  the 
error  that  caused  a failure,  removing  the 
error,  and  proving  that  the  failure  no 
longer  occurs. 

ftih/re  identif cation:  the  process  of  comparing 
test  results  against  program  requirements 
to  establish  the  failures  that  have  occurred. 

sojtmre  reliability:  probability  of  failure<free 
operation  in  a specified  environment  for  a 
specified  time. 
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APPENDIX  B 

SOME  DETAILS  RELATING  TO 
EXECUTION  TIME  AND 
CALENDAR  TIME  MODELS 

The  independence  of  one  program  failure 
from  another  postulated  in  Assumption  2 is 
met  by  most  programs.  There  are  very  many 
sources  and  reasons  for  erron,  involving 
different  persons  and  different  program  func* 
dons;  This  very  diversity  would  indicate 
independence.  However,  even  if  the  same 
error  source  is  causing  two  or  more  errors  (e.g., 
a typographical  error  in  a variable  name  in  two 
diflierent  places),  it  is  unlikely  that  the  failures 
resulting  from  these  errors  have  much  inter- 
dependence (i.e.,  one  causes  the  other). 

One  might  thiok  that  the  constant  average 
failure  occurrence  rate  mendoned  in  Assump- 
don  2 implies  that  ail  programmers  code 
equally  well  It  does  not.  The  implicadon 
would  be  necessary  only  if  the  code  executed 
. between  each  pair  of  failures  were  developed  by 
a single  and  different  designer.  Ordinarily,  the 
time  between  failures  is  sufliciendy  large  and 
the  program  structure  is  suffidendy  complex 
that  the  executed  code  will  have  b^  assod- 
ated  with  several  programmers.  Furthermore, 
these  programmers  can  usually  be  viewed  as 
consdtudng  a random  sample  from  the  popula- 
don  of  project  programmers.  Therefore,  the 
concept  of  a constant  average  failure  rate  does 
not  depend  on  idendcal  error  rates  of  individual 
designers. 

The  model  is  not  dependent  on  the  nature 
of  the  errors;  they  may  result  from  single  or 
multi-instnicdon  defects,  from  wrong  code, 
missing  code,  superfluous  code,  or  misplaced 
code.  Neither  is  it  dependent  on  the  cause  of 
the  errors  such  as  requirements  misimderstand* 
ing,  poor  interface  definition  or  communica- 
tioit,  poor  algorithm,  '^stupid”  error,  etc. 

It  wu  noted  that  Mo  represents  the  total 
number  of  failures  potsMe  during  the  main- 
tained life  of  the  software.  Whether  or  not  this 
number  is  reached  depends  on  the  extent  to 
which  the  software  is  operated.  After  the  end 
of  the  maintained  life  of  a program,  the 
number  of  remaining  errors  will  remain  con- 
stant, the  possible  number  of  failures  will  be 
unbounded,  and  the  MTTF  will  be  constant 
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The  resource  expenditure  parameters  of 
the  calendar  time  model  represent  average 
values.  It  is  assumed  that  the  project  for  which 
calendar  time  predictions  are  being  made  is 
large  enough  so  that  small-sample  effecu  of 
different  programmer  skill  levels  and  levels  of 
task  difficulty  may  be  ignored  (unless  they  are 
known  and  can  be  accounted  for).  Reasonable 
results  have  been  obtained  on  projects  as  small 
u S programmers  and  5000  instructions. 

It  may  further  illuminate  the  concept  of 
the  utilization  factor  to  consider  how  use  of 
seleeave  overtime  for  those  debuggers  who  have 
a failure  backlog  affects  the  calendar  time 
required  for  the  testing  effort.  Let  a be  the 
overtime  fraction  (ratio  of  overtime  to  standard 
time  for  each  work  day).  Now  the  effective 
available  personnel  are  increased  by  the  factor 
(I  a)  and  hence  the  calendar  time  for  the 
faiiure-correction-peraonnel*limited  segment  of 
the  test  phase  is  r^uced  by  this  factor. 

The  probability  for  each  debugger  that 
there  are  one  or  more  failures  awaiting  correc- 
tion or  being  corrected  is  [12).  Hence  the 
actual  work  required  is  increased  by  the  factor 
(l4-pfa).  For  example,  if  fir  0.2  and 
a » 0.2.  and  we  assiune  that  the  entire  test 
phase  is  failure-correction-personnel  limited, 
then  a 20  percent  improvement  in  calendar 
time  is  obtained  at  the  expense  of  4^  percent  in 
additional  work.  The  value  of  seieetive  overtime 
is  clearly  demonstrated. 

appendix  C 

HANDLING  DESIGN  CHANGES 

The  theory  developed  in  this  paper  did 
not  originally  account  for  design  changes  that 
might  occur  during  test,  causing  the  total  errors 
and  therefore  the  total  ponibic  number  of 
failures  to  increase.  One  could  develop  a more 
complex  model  than  the  piecewise  exponential 
to  handle  error  growth  (e.g.,  a WiebuU  model) 
but  this  would  result  in  a loss  of  intuitive  and 
analytical  simplicity.  Furthermore,  there 
would  be  more  parameters  to  be  estimated, 
probably  with  less  accuracy  possible  for  any 
given  sample  size.  Finally,  the  error  grosrth 
process  is  not  well  known,  hence  the  greater 
possible  precision  of  a more  complex  model 
does  not  seem  warranted  Three  approaches  for 


dealing  with  error  growth,  based  on  the  piece- 
wise  exponential  model,  will  be  described. 

The  first  approach  is  best  suited  to  the 
situation  where  there  are  a number  of  small 
changes  introduced  continually  throughout  the 
test  period.  In  this  case,  our  buic  hypothesis 
of  a constant  average  error  occurrence  rate  is 
approximated  satisfactorily  as  long  as: 

(a)  the  changes  are  well  distributed  across 
the  exeeuttOH  space  of  the  program  at  all 

(b)  the  size  of  the  changes  in  terms  of 
number  of  instructions  has  a sutistical 
distribution  that  is  time-invariant,  and 

(c)  the  code  in  the  changes  on  the  average  is 
debugged  to  the  same  degree  as  the  ori- 
ginal code  at  the  start  of  system  test. 

Experience  to  date  has  indicated  that 
these  assumptions  are  usually  satisfied: 

(a)  Distribution  of  changes  may  be  corre- 
lated with  program  function  and  there- 
fore. be  nonrandom  in  memory  space; 
however,  it  is  usually  not  localized  in 
execution  space  unless  the  function  in 
which  they  are  localized  also  happens  to 
be  highly  periodic. 

(b)  No  compelling  reason  for  assuming 
time-dependence  of  change  size  could  be 
demonstrated;  analogous  dau  on  effort 
required  to  correct  errors  [13]  shows  no 
time-dependence. 

(c)  Although  code  in  any  partictilar  change 
may  be  better  or  more  poorly  debugged 
than  the  original  code,  no  reason  could 
be  found  to  justify  a eonsistem  difference 
nor  does  experience  indicate  such  a 
difference. 

From  a large  number  of  simulations  it  hu 
been  determinod  that  maximum  likelihood 
predictions  of  Afo,  Tq,  and  other  quantities  dur- 
ing test  will  reflect  the  added  fulures  intro- 
duced by  design  changes  quite  accurately  as 
long  tt  the  ratio  AAfo/Afo  is  “small,"  where 
^Aio  is  the  number  of  failures  added  at  one 
time.  **Small"  means  that  ^o/^o  is  sbout 
0.25  or  less  in  the  case  where  the  changes  are 
modiflcadons  to  the  original  code  and  about  0.5 
or  less  where  the  changes  represent  additions  to 
the  code  (resulting  in  corresponding  increases 
to  the  total  size  of  the  program). 
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The  total  failures  Mq  can  be  related  to  the 
inherent  errors  at  the  start  of  test  /Vq  (and 
failures  experienced  m to  errors  corrected  n)  by 
appropriately  reducing  the  value  of  the  error 
(Auction  factor  B,  if  one  can  assume  that  the 
rate  at  which  erron  caused  by  design  changes 
are  introduced  is  proportional  to  the  program 
error  detection  rate.  This  assumption  implies 
(12,  Eq.  (2)  and  (12)1  that  the  rate  of  errors 
introdu^  by  design  changes  must  be  propor* 
tional  to  the  number  of  errors  remaining.  In 
many  cases,  this  is  a reasonable  model  of  real- 
ity, because  it  implies  that  the  rate  of  change 
decreases  as  testing  proceeds.  If  one  views 
design  changes  as  corrections  for  requirements 
errors,  then  the  above  model  is  followed  if  the 
ratio  of  requirements  errors  identified  to  pro- 
gram errors  identified  remains  constant 
throughout  the  test  period.  See  Appendix  0 
for  further  discussion. 

The  second  approach  applies  to  the  case 
of  a number  of  small  changes  well  distributed 
across  the  program  but  introduced  at  one  time; 
eg.,  with  a new  release  of  a program.  In  this 
case,  there  is  a “jump’*  increase  iNo  in  the 
total  errors  (and  a corresponding  increase  of 
lAio  in  the  total  possible  failures).  If  the  ratio 
is  smaU,  the  reesdmation  of  Mo- 
and  oth^  quantities  during  test  will  reflect  the 
increases  or  decreases  fairly  realistically,  as  pre- 
viously indicated. 

When  lAfo/Mo  is  large,  overshoots  (i.e.. 
overestimates  which  damp  out  with  time)  tend 
to  occur  which  increase  in  severity  with 
XMo/Mo.  The  overshoot  tends  to  be  greater 
for  modified  code  than  for  added  code.  It  tends 
to  be  worse  for  To  than  Mo  for  intermediate 
values  of  XMo/Mo,  but  the  effect  for  Mo  ulti- 
mately predominates  at  higher  values.  The 
effect  on  Mo  tends  to  increase  as  m (the 
number  of  failures  experienced  to  date  and 
used  in  the  estimation  process)  decreases;  the 
effect  on  To  tends  to  increase  u m increases. 

A sensible  approach  for  dealing  with 
ovenhoois  seems  to  be  to  focus  on  the  lower 
confidence  limit,  assuming  that  it  more  truly 
represents  the  state  of  the  project  at  this  point 
thu  the  maximum  likelihood  estimate. 

The  third  approach  should  be  used  when 
an  entire  subsystem  or  subsystems  are  added, 
where  the  subsystems  are  functionally  separated 


from  the  rest  of  the  program  so  chat  the  new 
failures  introduced  are  essentially  localized  (i.e., 
they  are  not  well  distributed  across  the  entire 
system).  In  this  case,  the  system  should  be 
divided  into  two  or  more  separate  subsystems. 
Work  with  each  separately,  and  then  combine 
reliabilities  in  accordance  with  reliability  com- 
binatorial rules.  One  example  of  this  situation 
is  an  application  program  that  is  added  to  a 
well-debugged  operating  system.  In  fact,  the 
operating  system  may  be  sufficiently  reliable 
that  any  failures  it  would  contribute  to  the 
overall  system  could  be  completely  ignored. 

It  should  be  noted  that  the  errors  in  each 
subsystem  will  probably  be  distributed  with 
different  average  occurrence  rates.  Hence  it  is 
necessary  to  continually  deal  with  each  subys- 
tem  separately,  even  if  they  are  viewed  as  being 
integrated  at  some  point.  Execution  times  and 
failures  must  be  allocated  to  each  subsystem. 
Note  that  to  allocate  execution  times,  one  must 
know  the  average  percentage  of  time  each  sub- 
system is  executing.  For  this  allocation  to  be 
valid,  the  intervals  for  which  each  subsystem 
runs  must  be  small  compared  with  the  failure 
intervals  so  that  average  execution  percentages 
can  be  validly  applied  (i.e.,  so  that  the  Central 
Limit  Theorem  applies). 

The  quantities  Mo  . To,  T/*,  Am  , Ar,  and 
Ar  are  estimated  separately  for  each  subsystem. 
For  the  estimation  process,  the  mean  time  to 
failure  objective  of  the  system  must  be  allo- 
cated among  the  subsystems  by  allocating  relia- 
bility objectives.  The  estimation  of  Ar  will 
require  an  allocation  of  resources  (failure 
identification  and  correction  personnel  and 
computer  time)  among  subsystems.  To  deter- 
mine overall  system  figures,  the  .Vfo's,  Am’s, 
and  Af’s  are  added,  the  to'i  and  tp's  are 
combined  using  standard  combinatorial  rules  on 
their  associated  reliabilities  (the  rules  applied 
depend  on  the  structure  that  relates  subsystem 
failures  to  over^l  system  failure),  and  the  max- 
imum of  the  It's  is  taken. 

The  approach  can  theoretically  be 
extended  to  any  number  of  subsystems.  How- 
ever, the  quality  of  sutistical  estimation  may 
suffer  at  some  point  if  the  subsystems  become 
too  small  to  yield  good  samples  of  failures. 

In  discussing  the  first  two  approaches,  the 
phrase  “distribution  of  changes  across  the  pro- 


445 


^LCM  WORKSHOP.  AUGUST  1977  . 


grain  has  been  used."  In  determining  which 
approach  to  employ,  note  that  the  distribution 
is  in  execution  space  rather  than  instruction 
space.  For  example,  the  changes  may  ail  be 
bunched  in  one  routine  but  if 

(a)  the  average  failure  interval  is  large 
enough  so  that  the  code  executed  during 
that  interval  consists  of  a large  number 
of  different  routines  (of  which  the 
changed  routine  is  one)  executed  a 
number  of  times  each,  and 

(b)  only  one  or  two  newly  added  errors  in  a 
particular  execution  of  the  changed  rou- 
tine are  activated  by  the  particular  state 
the  system  is  in  at  that  time  (usuaily 
true), 

it  will  be  seen  that  ttut  changes  and  the  errors 
they  introduce  are  well  distributed  in  execution 
space. 

It  should  be  noted  that  when  a change 
involves  added  code,  the  linear  execution  fre- 
quency /will  decrease.  This  must  be  accounted 
for  if  one  is  computing  the  error  exposure  ratio 
(ratio  of  error  exposure  frequency  to  error 
velocity)  K from  Tq. 

APPENDIX  D 

practical  ASPEiTTS  OF 
PARAMETER  ESTIMATION 

The  determination  of  the  amount  of  avail- 
able computer  time  must  take  into  account 
anticipated  preventive  and  corrective  mainte- 
nance times. 

The  prescribed  work  period  represents 
“available  time"  or  the  total  amount  of  time 
that  the  average  member  of  the  project  can 
work  upon  requesL  In  generaL  available  tune  is 
esublished  by  company  policy  or  the  practical 
coositteratioo  of  the  maximum  time  one  can 
reasonably  request  a person  to  work.  Normally, 
any  one  person  will  not  work  at  the  limit  for  an 
extended  period,  but  it  can  happen.  "Avail- 
able" also  implies  reasonably  short  notice.  It 
does  not  imply  that  the  project  member  must 
be  standing  by  for  the  available  time,  but  it 
does  imply,  for  example,  that  he  or  she  must  be 
able  to  work  two  hours  overtime  sometime 
prior  to  the  next  day,  when  requested  at  4 PM. 


The  best  estimate  of  the  number  of 
failure  correction  personnel  Pf  is  given  by  the 
number  of  programmers  available  who  were 
occupied  full  time  in  coding  the  various  new  or 
extensively  modified  units  of  the  system  under 
test 

The  parameter  P/  represents  the  number 
of  members  of  the  test  team,  or  the  number  of 
personnel  familiar  with  system  requirements 
and  test  plans  and  available  for  executing  tesu 
and  verifying  proper  operation  of  the  system  or 
identifying  departures  from  proper  operation. 

The  key  concept  in  resolving  questions  of 
interpretation  in  counting  resource  quantities  or 
requiremenu  under  special  conditions  that  may 
exist  on  different  projects  is  the  idea  of  a 
resource  being  limiting.  For  example,  support 
staff  (programming  librarians,  technical  writers, 
etc)  are  not  counted  because  they  do  not 
directly  constitute  a limiting  resource.  Of 
course,  the  amount  of  support  available  may 
affect  the  environment  in  which  the  project 
operates,  so  that  the  resource  parameters 
(tLc>  (si,  9c,  9f)  may  be  a function  of  the 
support  level.  That  is,  a Ugher  level  of  support 
may  reduce  the  amount  of  work  done  by  the 
failure  identification  and  failure  correction  per- 
sonnel for  each  failure  or  per  unit  execution 
time. 

The  failure  identification  activity  usuaily 
includes  the  generation  of  trouble  reports 
because  this  represents  a task  that  is  “inline" 
with  respect  to  identifying  a failure  and  thus 
limiting.  Tasks  associated  with  test  planning 
and  test  case  development  are  usually  (but  not 
always)  completed  prior  to  the  test  effort  and 
not  included.  The  failure  correction  activity 
includes  the  task  of  program  change  documen- 
tation, if  required  to  be  completed  during  test 
rather  than  after. 

It  is  assumed  that  each  person  involved  in 
failure  identification  can  run  and  examine  the 
results  from  any  test  that  is  made.  Failure 
correction  peronnel  are  assumed  to  work  only 
on  failures  that  have  been  attributed  to  their 
respective  design  areas.  If  a person  has 
sufficient  background  to  work  at  either  failure 
identification  or  failure  correction,  depending 
on  where  staff  is  required,  and  is  permitted  to 
do  so,  that  person  should  be  included  in  both 
counts.  This  docs  not  represent  duplication 
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because  only  oae  resource  at  a time  can  be  lim- 
iting. The  theory  and  programs  based  on  it 
view  the  personnel  numbers  as  available  per- 
sonnel; a person  is  not  “assigned”  simultane- 
ously to  two  functions. 

On  some  projects,  separate  failure 
identification  and  failure  correction  personnel 
are  not  used;  ue.,  each  person  does  both  jobs. 
Since  these  resources  are  essentially  merged, 
one  must  merge  them  and  the  resource  require- 
ments as  far  as  the  calendar  time  model  is  con- 
cerned. The  beat  way  to  do  this  is  to  first  set 
*^0;  this  will  cause  the  CF  and  FT  transition 
points  in  (12)  between  different  periods  of 
resource  limitation  to  occur  at  negative  values 
of  MTTF  (i.e,  be  nonexistent)  and  the 
integrand  for  the  failure-correction-personnel- 
limited  period  to  be  zero.  Thus  there  will  be  at 
most  one  transition  point,  IC,  and  failure 
correction  work  required  cannot  by  iaelfbe  lim- 
iting. The  resource  requirement  n/  should  be 
set  to  a Mr  *1*  • The  value  of  P/  should  be 

taken  as  the  total  number  of  available  program- 
ming personneL  The  values  set  for  /V  and 
will  be  immaterial  since  there  can  be  no 
faflure-correction-personnei  limited  period.  Oo 
not,  however,  set  either  one  to  zero,  because 
this  will  lead  to  undefined  results  for  the  transi- 
tion points  and  the  integrand  indicated  above. 

A relatively  painless  approach  to  collec- 
tion of  the  dam  for  the  estimation  of  the  debug 
environment  parameters  iLf.  , and  9/  is  to 
obtain  weekly  resource  expenditure  totals  from 
each  person  involved  in  testing.  Note  that  time 
spent  in  proving  suspected  failures  false  is 
included  and  thus  wiU  be  apportioned  among 
real  failurea.  In  cases  where  only  one  person  is 
identifying  the  failures,  the  data  gathered  will 
be  Very  individual-dependent  unless  we  average 
data  for  several  “one  identifier”  projects 
involving  different  persons.  An  appropriate 
weighting  function  for  the  daa  in  making  the 
least  squares  fit  (13)  is  probably  the  square  root 
of  the  execution  time  associated  with  each  dau 
point,  except  for  the  failure  correction  parame- 
ter fit,  which  should  use  the  square  root  of  the 
number  of  failures  actually  corrected.  Over- 
head such  as  vacations  and  absences,  training 
and  administrative  activities  should  be  included. 
Usually  measurements  made  of  the  parameters 
will  represent  net  times;  they  must  be  increased 


by  a suitable  overhead  factor.  A typical  value  is 
1.35. 

It  should  be  noted  that  poor  computer 
turnaround  time  can  affect  the  pace  at  which 
failures  are  corrected.  If  the  total  nonproduc- 
tive wait  time  involved  in  correcting  each 
failure  is  substantial  in  comparison  with  average 
failure  work  time  required  per  failure  then 
should  be  increased  by  the  ratio  of  failure 
correction  work  time  plus  wait  time  to  failure 
correction  work  time.  Note  that  there  is  usu- 
ally only  one  approach  to  the  machine  for  each 
failure  identification;  hence  turnaround  time 
has  little  effea  on  that  process.* 

If  the  errors  causing  some  of  the  failures 
caimot  be  determined,  the  jxc  and  Mr  parame- 
ten  that  would  ordinarily  be  employed  must  be 
adjusted  by  multiplying  by  the  detecubility  ratio 
D (see  below)  to  account  for  the  reduced 
amount  of  resources  required. 

Although  the  error  reduction  factor  B is 
.not  an  essential  parameter  of  the  software  relia- 
bility modeU  knowledge  of  it  is  necessary  if  one 
wishes  to  relate  failures  and  errors.  It  may  be 
computed  from 

B -D(l -t- A)(l  - C)  . (Dl) 

The  parameter  D is  the  detecubility  ratio  or 
proportion  of  failures  whose  errors  can  be 
found.  It  is  usually  I oo  development  projecu 
but  ofun  less  than  that  in  compuution  cenur 
operation.  The  quantity  A is  the  associability 
ratio,  the  ratio  of  errors  discovered  by  reading 
during  test  to  errors  discovered  by  testing 
proper.  Discovery  of  errors  during  test  by 
reacting  is  usually  the  result  of  associating  an 
error  discovered  by  testing  with  closely  related 
errors.  The  parameter  G is  the  error  growth 
ratio  or  the  increase  in  erron  per  error 
corrected.  This  factor  can  account  for  erron 
spawned  in  correcting  other  errors  and  for 
erron  added  by  design  changes  (see  Appendix 
C).  When  error  growth  occun,  No  must  be 
interpreted  as  the  initial  inherent  errors  at  the 
start  of  test. 


*Slioomaa  and  Boteky  (1]|  Invn  coUacttd  dau  that  in- 
dicau  an  avtnea  of  0.61  nina  par  failurt  for  failura 
idannllcation  and  l.JS  runs  par  faiteira  for  failure 
corraeiion. 
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At  present,  the  values  of  A.D^  and  G 
must  be  determined  by  uking  dau  for  the  par- 
ticular project  or  using  values  from  a similar 
^ject  Probably  some  or  all  of  these  parame- 
ters will  eventually  either  be  found  to  be  con- 
stant or  to  be  determinable  functions  of  the  test 
or  operating  environment.  Dau  taken  by 
Miyamoto  [4,  p.  199]  and  his  associates,  inur- 
preted  in  terms  of  the  author’s  formulation  of 
G for  spawned  errors  only,  would  indicate 
values  of  G fhim  0.05  to  0.09  over  a number  of 
cases  of  development  of  general  purpose 
software  systems.  This  compare  with  values 
from  0 to  0.06  over  the  four  projecu  reported 
in  this  paper. 


Now  % is  given  by 


fo-Cr,  1 


(E5) 


where  is  the  sample  mean  of  time  to  failure 
during  test 

The  variance  of  i is  given  by 


vor  (i) 


_J 


+ 


(E6) 


where 


APPENDIX  E 

PROGRAM  PARAMETER 
REESTIMATION  DETAILS 

The  important  relationships  in  the  max- 
imum likelihood  estimation  process  are  given 
below.  For  derivations,  see  (12,  Appendix  Bl. 

We  find  Mo  implicitly  from 

i '"I  - 

where  d is  the  failure  moment  statistic 


A#'  - ♦‘(Mo  + 1)  - d'CMo  + I - m)  (E7) 

and  ♦*  is  the  trigamraa  function  (18,  p.  44l. 
Confidence  intervals  nuy  be  established  for  Mo 
by  determining  the  values  of  Mq  that 
correspond  to  values  of  ♦ chosen  so  that 

♦ - ♦ ± S.D.  (♦)  . (E8) 

where 

SID.fd) -[vor  (E9) 


♦ 


m 

I 


(/ 


- Dt,'  . 


(E2) 


in  which  the  are  the  execution  time  intervals 
between  failures  and  r.  is  the  total  execution 
time.  Now 


(E3) 


and  ? is  the  confidence  level.  The  inurvals 
are  based  on  the  application  of  Chebyshev’s 
inequality.  Although  this  inequality  tends  to 
produce  conservatively  large  intervals,  it  is 
probably  best  to  employ  it  until  more  experi- 
ence hu  been  gained  with  the  reliability  model 
of  this  paper. 

The  variance  of  fo  is  given  by 
Ti 

w (fo)  - — . (ElO) 

m 


where 

Ad  * djMo  ij  ~ + 1 “ (E4) 

and  ♦ is  the  psi  (digamma)  function  (181. 


Confidence  intervals  for  Tq  are  found  by  apply* 
ing  (E8)  with  ♦ replaced  by  fo . 

Experience  indicates  that  the  quality  of 
estimation  can  be  considerably  improved  for 
small  sample  sizes  by  smoothing  ♦ . On  the 
other  hand,  the  smoothing  should  eventually  be 
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removed  because  it  impairs  the  responsiveness 
of  the  estimation  algorithms  as  the  number  of 
remaining  errors  becomes  small.  After  trying  a 
number  of  different  smoothers,  a variable 
length  smoother  which  builds  up  to  a length  of 
40  samples  at  m * 40  and  then  drops  to  a 
length  of  one  sample  (i.e.,  no  smoothing)  at  m 
• 79  was  adopted.  The  smoother  weights  each 
value  of  ^ by  the  corresponding  /n 

It  has  been  found  convenient  to  take  run 
time  dau  in  a log  (the  logging  could  be 
automated).  A log  could  also  be  used  for 
recording  (possibly  interactively)  any 
occurrence  which  might  be  a failure.  Alterna- 
tively, trouble  report  forms  could  be  used.  If 
there  are  multiple  testers,  it  is  necessary  to 
accurately  record  the  time  of  day  of  each  test 
and  set  up  a procedure  so  that  the  test  logs  and 
trouble  dau  may  be  conveniently  interleaved  in 
time  sequence.  Errors  found  by  a programmer 
without  test  (by  code  reading  or  other  means) 
during  the  test  period  should  not  be  counted  as 
failures. 

Observation  of  failures  may  occur  at  the 
time  of  program  operation  or  when  analyzing 
program  output  afterwards.  Therefore,  it 
proved  best  to  allow  one  or  two  days  to  elapse 
after  a test  run  before  using  the  failure  interval 
data,  so  that  false  failures  could  be  weeded  out 
and  the  results  thoroughly  analyzed  to  minim- 
ize missed  failures.  Experience  indicates  that, 
after  analysis,  one  is  more  likely  to  miss 
failures  than  to  report  false  ones.  If  a failure 
reoccurs  before  the  error  that  caused  it  can  be 
fixed  (assuming  that  the  error  could  be  deter- 
mined), then  the  repetition  of  the  failure 
should  not  be  counted. 

Although  our  experience  indicated  that 
perhaps  9S  percent  or  more  of  failures  could  be 
readily  clashed  on  examination  as  software  or 
nonsoftware  (hardware,  operator,  etc.),  occa- 
sionally the  choice  became  uncertain.  The  cri- 
terion that  was  followed  was  to  classify  the 
failure  as  nonsoftware  if  it  could  not  be  made 
to  recur  when  the  program  was  rerun  with  all 
software  and  with  known  nonsoftware  inputs 
exactly  the  same. 

Almost  ail  failures  could  be  readily  associ- 
ated with  particular  test  times;  those  that  could 
not,  did  relate  to  a fairly  short  time  interval.  A 
randomly  selected  value  within  the  interval  was 


taken  as  the  time  of  failure  in  those  cases.  A 
deliberate  decision  during  the  maintained  life  of 
the  program  that  an  error  will  not  be  fixed  is 
equivalent  to  a redefinition  of  the  requirements 
so  that  the  former  “failure”  no  longer  exists  as 
such  and  should  no  longer  be  counted. 

If  there  are  very  few  failures  (e.g.,  40  or 
less)  during  the  test  phase,  the  confidence 
intervals  may  be  very  broad  due  to  the  small 
sample  size.  Somewhat  narrower  intervals  but 
a worst  case  prediction  may  be  obtained  by 
assuming  that  a failure  occurs  right  at  the  end 
of  testing. 

If  it  is  difficult  to  measure  actual  CPU 
time  for  the  failure  intervals,  one  can  measure 
another  quantity  that  is  proportional  to  CPU 
time,  on  the  average.  One  possibility  is  elapsed 
test  time.  As  long  as  CPU  utilization  is  approx- 
imately the  same  over  different  failure  inter- 
vals, the  results  should  be  satisfactory.  Note, 
however,  that  MTTF  values  are  now  specified 
in  terms  of  the  new  quantity.  This  may  be  an 
advantage  if  the  quantity  is  a natural  one  for 
the  application.  However,  adiustment  of  the 
objective  MTTF  Tf  may  be  necessary.  The 
linear  execution  frequency  / of  the  program 
with  respect  to  the  new  quantity  will  probably 
be  lower,  as  will  the  parameters  9c  and  9/. 

If  the  ratio  of  proportionality  should  differ 
for  any  reason  for  different  parts  of  the  system 
test  period,  failure  intervals  should  be  adjusted 
so  they  are  all  given  in  common  terms.  For 
example,  if  the  quantity  measured  changes 
from  l\r  to  liT  , then  one  can  adjust  to  the  ini- 
tial type  of  measurement  by  adjusting  the  latter 
intervals  by  the  ratio  f)//;  . The  original 
/•  • snd  9/  will  not  change. 

If  testing  starts  before  integration  is  com- 
plete, then  there  may  be  periods  of  inactivity  in 
testing  and  debugging  due  to  waits  for  programs 
not  yet  ready.  These  periods  must  be  estimated 
separately  and  added  to  estimates  of  remaining 
t or  else  t must  not  be  viewed  as  “running” 
when  they  occur.  Note  that  severe  computer 
outages  (one  day  or  more)  and  delays  due  to 
requirements  changes  have  not  been  included 
in  the  estimate  of  remaining  test  phase  length 
and  must  be  added  if  expected. 

If  it  is  desired  to  determine  the  sutus  of  a 
program  that  is  released  to  the  field  with  some 
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errors  not  fixed,  back  up  in  the  test  phase  by 
the  number  of  failures  corresponding  to  the 
errors  outstanding  and  run  the  prediction  pro* 
gram  at  this  point 

Experience  has  generally  indicated  that  it 
is  best  to  ignore  severity  in  estimating  the  pro* 
gram  parameten  and  testing  progress  quantities. 
It  is  better  to  develop  MTTFs  for  each  failure 
class  by  dividing  the  overall  MTTF  by  the  frac* 
tion  of  failures  occurring  in  that  class.  This 
method  takes  advantage  of  the  largest  possible 
sample  size  and  avoids  the  problem  of  alloca- 
tion of  MTTF  objectives  and  resources  to 
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1.  Introduction 

A major  problem  in  managing  large-scale  software  projects  has 
traditionally  been  the  difficulty  of  measuring  the  quality  of  programs. 
Despite  the  wealth  of  published  material  advocating  methodologies  for 
Improving  software  quality » their  effectiveness  has  usually  been  judged 
subjectively  rather  than  quantitatively.  Reliability  models  which 
describe  the  frequency  of  failure  of  software  system  by  a distribution 
whose  parameters  can  be  estimated  empirically,  provide  one  means  of  making 
these  comparisons.  The  empirical  measurement  of  program  quality  In  the 
form  of  reliability  Is  also  an  important  Ingredient  for  any  attempt  to 
explain  the  patterns  which  appear  to  govern  the  long-term  development  of 
large-scale  software  projects. 

Since  the  reliability  techniques  of  hardware  systems  had  for 
some  time  been  successfully  used  to  Improve  designs  and  project  management 
methods.  It  Is  not  surprising  that  similar  reliability  models  have  been 
proposed  for  understanding  the  failures  of  software  systems.  Most  of 
these  models  have  been  of  the  bug-counting  type.  They  make  little  or  no 
analysis  of  the  Internal  structure  of  the  software  being  considered.  Future 
Instantaneous  failure  rates  are  predicted  on  the  basis  of  past  failure 
history.  The  central  Idea  behind  such  models  Is  that  a computer  program 
will  at  any  Instant  contain  a fixed  number  of  errors.  As  the  program 
executes  In  Its  operational  environment  these  bugs  manifest  themselves  In 
the  form  of  sporadic  system  failures  whose  frequency  Is  taken  to  be 
proportional  to  the  number  of  bugs  reoialnlng  In  the  program.  When  a 
failure  Is  observed  efforts  are  made  to  remove  Its  cause  resulting  In 
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removal  of  one  error  from  the  program  after  some  fixed  average  number  of 
failures  In  the  case  of  models  In  (4)  and  (7) , and  In  the  case  of  the 
model  In  (3)  a stochastically  characterized  jump  In  the  number  of  errors 
remaining  In  the  program.  The  effect  of  these  models  Is,  very  roughly, 
to  predict  that  the  frequency  of  failure  of  software  system  decays  In  an 
approximately  exponential  fashion  over  time.  This  Is  the  result  of  the 
error  removal  process  proceeding  at  a rate  proportional  to  the  nuniber  of 
errors  remaining.  Using  this  decay  law  one  can  estimate  the  Initial 
number  of  errors  In  the  program  from  looking  at  Its  past  performance  and 
also  predict  Its  future  reliability. 

These  reliability  models  have  been  tested  on  actiial  failure 
data  from  some  operating  systems  and  hardware  system  controllers  giving 
predictions  which  were  clearly  of  management  value.  However,  there  remain 
considerable  problems  In  Interpreting  these  results  and  even  more  so  In 
extending  the  theory  to  deal  with  other  types  of  software  system  not  so 
directly  concerned  with  real-time  control.  This  paper  explores  the 
apparent  limitations  of  the  present  methods  and  makes  some  suggestions 
for  refining  them. 

Most  of  the  problems  with  this  approach  to  software  reliability 
stem  ultimately  from  the  shallowness  of  the  analogy  between  software  and 
hardware  systems.  Hsrdware  reliability  theory  Is  concerned  with  the 
period  of  time  which  elapses  before  sn  atomic  system  component  becomes 
degraded  and  falls.  The  systems  are  repaired  after  failure  by  replacing 
the  bad  component  with  an  Identical  (but  working)  copy.  The  lifetime  of 
atomic  components  can  be  described  probabilistically,  san^llng  tiste-untU- 
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failure  from  a large  ntoiber  of  Identical  coplea  of  the  part  will  determine 

the  distribution.  The  distribution  of  tlme-untU-f allure  of  a hardware  ^ ‘ 

system  Is  In  principle  Just  a complex  combinatorial  calculation  using  the  ^ 

distributions  of  the  lifetimes  of  Its  components.  Contrast  this  with 

software  systems  where  there  Is  no  degradation  over  time  and  where  repair  i 

Is  not  by  replacement.  There  Is  no  probabilistic  model  to  describe  the 

failure  behaviour  of  atoms  of  software.  Furthermore*  the  errors  which 

cause  unreliability  In  software  are  errors  of  design  - a problem  not 

1 

usually  tackled  In  the  analysis  of  hardware  reliability.  Also  hardware 
errors  have  a definite  location  whereas  software  errors  can  be  an  Incon- 
sistency between  several  sectlona  of  the  code. 

Further  examination  of  the  problems  arising  from  this  approach 
to  software  reliability  is  divided  Into  three  sections.  Section  2 shows 
that  a necessarily  probabilistic  model  for  the  reliability  of  an  essentially 
deterministic  software  system  can  be  constructed  only  with  careful  con- 
sideration of  Its  Input  space  and  usage.  Section  3 concentrates  on  the 
special  problems  of  analysing  the  reliability  of  programs  which  have 
complex  Interfaces  with  other  software.  Section  4 takes  a life  cycle 
approach  and  considers  the  effects  of  maintenance  and  system  redesign  on 
reliability. 


The  Input  Space  and  Paage  Conalderatlons 


r 
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The  primary  objective  of  a software  reliability  theory  is  to 
construct  a model  which  describes  probabilistically  the  occurrence  of 
failures  of  a prograo  so  that  the  likelihood  of  any  future  pattern  of 
failure  behaviour  can  be  predicted.  Predictions  of  this  nature  are 
useful: 

(1)  in  managing  the  system  which  uses  the  software; 

(11)  in  managing  the  repair  and  maintenance  of  the  software 
itself;  and 

(ill)  in  guaranteeing  the  quality  of  the  software  before  It 
la  actually  put  Into  live  use. 

Predictions  for  all  three  uses,  but  most  particularly  (111),  all  depend 
on  the  assumption  that  there  Is  no  significant  shift  In  the  distribution 
of  Inputs  presented  to  the  system  during  the  period  of  time  under  consi- 
deration* In  analysing  the  reliability  of  hardware  systems  this  assumption 
Is  easily  Justified.  Althou^  the  lifetimes  of  some  hardware  components 
can  be  Influenced  by  the  extent  to  uhlch  the  component  has  been  used  and 
stressed,  the  class  of  Inputs  having  this  effect  Is  not  precisely  defined 
and  unpredictable  random  effects  still  dominate  the  process.  In  contrast 
to  this  there  Is  no  reason  In  principle  (although  It  may  be  extremely  hard 
to  do  In  practice)  Why  the  input  space  of  a program  cannot  be  partitioned 
into  those  for  which  it  works  and  those  for  which  It  falls  since  Its 
behaviour  Is  repeatable  whenever  the  Input  Is  exactly  reproduced.  So  the 
reliability  of  a software  system  depends  very  directly  on  the  Inputs  with 
which  it  is  driven.  If  some  probabilistic  measure  of  software  reliability 
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is  to  b«  given*  then  the  only  means  by  which  stochastic  festtires  can 
enter  the  model  Is  In  the  form  of  a probabilistic  description  of  the 
process  by  which  input  data  Is  selected.  Now  for  certain  types  of 
systems  this  Implicit  assumption  seems  quite  natural  - for  example  when 
building  a software  controller  for  a hardware  device  whose  operational 
characteristics  are  fixed  and  can  be  reasonably  treatad  as  stochastic 
fluctuations  around  some  steady-state  signal.  However*  moving  away 
from  the  real-time  embedded  applications  there  are  many  other  types  of 
software  for  which  this  assumption  seems  quite  inappropriate.  To  hope 
to  obtain  a reliability  theory  for  such  software  or  even  to  delineate 
those  applications  for  which  the  bug-counting  approach  can  hope  to  make 
useful  predictions  of  failure  rates*  one  must  Identify  those  qualities 
of  the  usage  environment  which  cannot  be  characterized  probabilistically 
and  yet  have  a significant  effect  on  the  failure  behaviour  of  the  system. 
Just  as  performance  analysis  of  computer  systems  has  depended  on  careful 
examination  and  characterization  of  workload  patterns*  a complete  software 
reliability  model  should  Include  an  analysis  of  the  form  of  Its  workload. 

In  arguing  for  a software  reliability  theory  which  la  more 
directly  based  on  usage*  one  Is  forced  to  admit  that  even  to  specify  In 
detail  what  constitutes  the  Input  space  of  a large-scale  software  system 
may  ba  a profoundly  daunting  task.  This  Is  particularly  true  of  real-time 
environments  where  sophisticated  linguistic  tools  are  reqtilred  to  describe 
all  possible  signal  histories  with  which  the  systom  nay  be  faced.  But 
despite  the  difficulty  of  obtaining  a complete  formal  specification  of  the 
allowable  usage  of  a large  systom*  most  software  developers  recognize  that 


they  do  have  some  measure  of  control  over  the  observed  failure  rates 
of  their  systems.  At  the  most  simple  level  It  Is  standard  practice  to 
test  new  software  with  a sequence  of  Input  cases  which  are  generally 
understood  to  be  increasingly  stressful.  The  rationale  Is  to  get  the 
easy  errors  out  first.  Such  an  approach  la  a clear  recognition  of  the 
fact  that  by  adjusting  the  usage  of  software  one  can  bias  the  mean 
frequency  of  failure  observed. 

One  desirable  advantage  of  MDSA's  reliability  analysis  over  the 
other  bug-counting  models  can  be  viewed  In  this  light.  This  Is  the  sug- 
gestion that  In  measuring  and  predicting  the  Intervals  between  program 
failures,  total  execution  time  should  be  used  rather  than  elapsed  actual 
time.  This  amounts  to  the  observation  that  the  Intensity  of  testing  and 
hence  the  observed  failure  rate  of  a software  system  can  be  adjusted  very 
directly  merely  by  performing  tests  at  a different  rate.  An  operating 
system  which  Is  run  for  only  12  hours  per  day  or  a suite  of  batch  programs 
executed  less  often  than  usual  can  be  expected  to  show  a reduced  rate  of  ^ 
failure.  Building  this  effect  Into  a description  of  the  error  removal 
process  gives  rise  to  a more  widely  applicable  model  for  reliability.  A 
theory  which  attempted  to  predict  the  elapsed  Interval  between  failure  of 
an  application  program  which  was  rm  only  occasionally  at  the  idilm  of 
particular  programmers  would  be  grappling  with  the  additional  problem  of 
modelling  human  actions. 

Another  way  of  altering  the  observed  rate  of  failure  Is  to 
specifically  search  for  Input  cases  lAlch  are  particularly  likely  or  unlikely 
to  create  problems  for  the  software.  This  factor  Is  specially  relevant  when 


the  behaviour  of  new  software  during  development  testing  and  dabugglng 
la  used  to  predict  Its  performance  In  operation.  Host  developing  software 
contains  errors  which  can  actually  be  found  by  Informal  desk  checking. 

Since  no  execution  time  with  the  actual  system  Is  consumed  In  detecting 
errors  by  this  method*  carrying  this  form  of  debugging  would  generate 
arbitrarily  short  Intarvals  of  execution  time  between  each  observed  failure 
and  correspondingly  low  reliability.  Although  purely  analytical  checking 
would  be  a complately  Impractical  way  for  validating  large-scale  software 
there  la  a continuum  of  approaches  lying  between  abstract  correctness 
proving  and  slmple-mlndad  random  selactlon  of  Input  data  for  test  cases. 

In  practice  It  Is  often  possible  to  Identify  dangerous  areas  In  a system. 

For  exaag>le  there  may  be  some  particularly  dangerous  Interfaces  or  cosplex 
control  flows  which  are  likely  to  contain  a hlghsr  density  of  errors  thsn 
the  rest  of  the  code.  Also  there  may  be  quite  specific  ways  In  which  a 
program  la  suspected  of  behaving  Incorrectly  (l.e.»  not  as  the  designer 
Intended)  but  which  can  be  conveniently  shown  to  be  errors  only  by  executing 
with  particular  tast  cases.  These  examples  are  concerned  with  achieving  a 
speeded-up  failure  rate.  Conversely  In  the  early  stages  of  system  Inte- 
gration It  Is  oftan  helpful  to  select  Input  data  which  avoids  araas  of 
difficulty  In  the  program  In  order  to  achieve  a partially  working  system 
as  quickly  as  possible. 

Vftwt  Is  meant  by  a continuum  of  actlvltlas  between  formal  correct- 
ness proving  and  tasting  with  arbitrarily  salected  data  can  be  made  more 
precise  by  referring  to  symbolic  execution  systems  ( ) ( ).  These  systans 
Interpret  programs  performing  algebraic  manipulations  on  exprasslons  for 
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the  data  rather  than  evaluating  variables  as  Is  done  during  notnal  execution. 
So  for  a program  with  no  data  dependent  path  decisions  the  output  variables 
computed  are  expressed  as  formulae  In  the  Input  variable  values,  enabling 
the  program  to  be  completely  verified  by  one  symbolic  execution.  Where 
there  are  data  dependent  path  decisions  to  be  made  the  program  tester  may 
choose  whether  to  proceed  theoretically,  suggest  loop  Invariants  and  try  to 
obtain  a complete  correctness  proof,  or  whether  to  Instantiate  state  variable 
expressions  with  enouj^  actual  data  to  allow  the  Interpretation  to  continue 
more  as  a test  execution.  By  choosing  between  specific  and  general  analysis 
independently  at  each  decision  point  In  the  symbolic  execution,  the  tester 
selects  a level  of  abstraction  for  his  validation  from  a wide  choice.  But 
this  use  of  hlgh'lnv*!  predicates  and  Inductive  arguments  In  validating 
programs  Is  just  a formalization  of  the  practice  of  thinking  about  the 
structure  and  semantics  of  the  code  In  order  to  adjust  the  rate  of  extracting 
errors.  The  formalization  offered  by  symbolic  execution  Is  useful  because 
It  suggests  ways  In  which  this  aspect  of  the  Intensity  of  testing  could  be 
measured. 

Besides  this  quite  general  testing  concept,  there  are  other  ways 
In  which  the  stressfulness  of  testing  can  be  adjusted  applicable  to  specific 
types  of  software.  Scientific  and  nwerlcal  computation  Is  an  example 
requiring  special  treatment.  Concepts  such  as  stiffness  of  differential 
equations  and  poor  conditioning  of  auitrlces  are  well  established.  The  re- 
quirements on  a suite  of  programs  to  process  nwerlcal  data  may  very  reason- 
ably require  that  results  are  computed  to  a given  accuracy  or  tolerance. 

The  reliability  estlmata  of  such  a suite  would  be  quite  meaningless  xmless 
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Infonatlon  w«r«  also  kapt  daacrlblng  how  much  poor  condltloalng  thara 
had  baan  in  tha  input  straaa  which  ganaratad  tha  oparatlonal  data  usad 
in  tha  nodal. 

Mowing  away  fron  aathanatieal  aoftwara  thara  ara  also  funda- 
aantal  dioicas  to  ba  aada  in  salacting  a tast  stratagy  for  systans 
aoftwara  which  is  intended  to  run  in  a wide  (often  potentially  infinita) 
warlaty  of  configurations  and  installations.  Typically  this  sort  of 
software  will  contain  a syaten  genaratlon  facility  which  will  put  to- 
gether an  appropriate  version  of  tha  systaa  whan  given  a description  of 
an  Intended  anvironnant.  In  evaluating  tha  quality  of  this  software 
systan  soaa  arbitrary  balance  is  struck  between  tasting  the  reliability 
of  particular  generated  versions  ruonlng  in  their  anvironnants  and 
tasting  the  reliability  of  systaa  generation  across  all  possible  environ- 
aants.  These  seen  to  give  two  entirely  different  concepts  of  what  is 
aaant  by  reliability  and  one  would  expect  quite  different  and  Indapandent 
aean  tine  between  failure  behaviour.  Tharaforet  prasuaably  by  adjusting 
tha  tasting  stratagy  any  Intaxaadlata  value  in  a cartaln  range  of 


reliability  could  ba  daaonstratad. 

A final  point  concerns  tha  variability  of  tasting  executions.  . 
Well-built  coeputar  software  behaves  detaralnlstleally  in  processing  each 
set  of  input  data.  By  repeating  a successful  (unsuccessful)  test  one 
can  bias  the  observed  reliability  of  the  systaa  upwards  (downwards).  Dis- 
regarding tha  raaota  possibility  of  this  sort  of  aaasure  being  intention- 
ally aanlpnlated*  one  still  has  to  face  the  fact  that  (after  a certain 
stage  in  debugging)  a prograa  usually  perforna  better  on  alddle-of-the-road 
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standard  Input  cases  than  In  exceptional  condition.  Some  rough  measure 
of  the  extent  to  which  test  Inputs  and  the  execution  environment  have 
ranged  over  all  possibilities  Is  therefore  needed.  Past  theoretical 
work  In  examining  the  completeness  of  various  testing  strategies  has 
producsd  only  limited  restilts  (5).  One  reason  for  this  Is  that  It  has 
concentrated  entirely  on  the  characterization  of  tests  by  means  of  the 
control  paths  which  they  exercise.  This  Is  concsptually  equivalent  to 
making  reconsaendatlons  for  the  testing  of  totally  unstructured  progrsms 
composed  of  erbltrary  control  Juoip.  But  much  of  the  debate  In  program- 
ming methodology  of  the  last  decade  has  resolved  that  unstructured 
progrsms  are  unmanageable  and  almost  csrtalnly  untestable.  It  follows 
that  progress  In  understanding  the  completeness  of  testing  strategies 
will  occur  only  idien  program  structure  Is  fully  taken  Into  account. 

In  susnary  we  have  Identified  a number  of  simple  decisions 
about  the  usage  of  software  which  could  be  taken  by  management  and  which 
would  Influence  the  observed  failure  rate.  Bug^countlng  based  modelling 
of  reliability  shotild  be  carried  out  only  In  environments  where: 

(1)  the  distribution  of  usage  has  been  fixed  once  for  all 
time  In  these  respects:  or 

(11)  the  relative  streasfulness  of  the  different  environments 
from  which  failure  data  has  been  collscted  can  be 
estimated  and  taken  Into  accoiat. 

This  second  condition  Is  particularly  relevant  In  predicting  the  Inatelled 
behaviour  of  a system  on  the  basis  of  Its  failure  record  during  testing 
snd  debugging  In  sn  artificial  software  development  environment. 
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3.  Problam«  Arising  from  the  Distributed  Nature  of  D««lgn  Errors 


In  the  preceding  section  It  wss  observed  that  the  primary 
function  of  a software  reliability  measure  Is  to  predict  future  progrsm 
failure  rates » and  that  In  such  predictions  account  most  be  taken 

of  possible  differences  In  system  usage  between  the  testing  period  for 
tdilch  failure  data  waa  collected  and  the  operational  period  for  which 
failure  rates  are  predicted.  Since  we  do  not  at  present  know  how  to 
evaluate  the  relative  atressfulness  of  different  execution  environments 
the  obvious  way  of  obtaining  good  reliability  predictions  for  a program 
Is  to  execute  It  with  data  as  representative  as  possible  of  the  ei^ected 
execution  envlromsent.  This  can  sometimes  be  achieved  before  Installation 
by  testing  newly  developed  software  live  In  Its  operational  environment » 
but  In  many  other  cases  such  an  approach  would  be  Impractical.  For  example 
ths  actual  environment  may  not  yet  be  built  or  there  may  be  danger  of  the 
newly  developed  software  severely  damaging  It.  This  section  explores  the 
problems  of  predicting  the  reliability  of  a software  system  with  a bug- 
counting  model  on  the  basis  of  Its  performance  In  a simulated  environment. 

Although  bug-counting  models  attempt  to  characterize  reliability 
la  terms  of  the  number  of  errors  remaining  In  a program,  the  Input  data 
on  which  they  ere  baaed  and  the  output  phenomena  predicted  are  both  ex- 
pressed In  terms  of  system  failure.  This  term  therefore  requires  some 
scrutiny.  Now  It  Is  usually  assumed  that  there  Is  some  well-defined  set 
of  requirements  lAlch  the  software  system  is  trying  to  satisfy  and  that 
It  Is  contravention  of  these  which  constitutes  failure.  The  problem  Is 
that  these  requirements  are  usually  stated  as  loose  assertions  about  the 
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behaviour  of  the  complete  system  - software  Interacting  with  Its 
environment.  When  the  program  being  evaluated  Is  an  operating  system 
or  device  controller  then  a failure  can  be  defined  as  a crash  or  loss 
of  response  of  the  system.  This  Is  a relatively  easy  case  since  the 
failures  can  be  roughly  devlded  Into  those  catised  by  hardware  or  operator 
malfunction  as  opposed  to  software  errors.  A rather  more  subtle  charac- 
terization of  failure  would  be  required  In  estimating  the  reliability 
of  a new  software  system  which  for  example  was  required  to  Interact  with 
an  existing  data  base.  The  problem  Is  that  for  an  Interface  between  the 
old  end  new  software  of  the  complexity  typically  arising  In  practice  no 
set  of  requirements  on  the  new  module  to  be  developed  could  hope  to  com- 
pletely characterize  Its  input/output  behaviour.  Among  the  choices  left 
open  there  are  likely  to  be  actions  which  alter  the  reliability  of  the 
Interconnection  of  the  systems. 

Error  counting  analysis  attempts  to  deal  with  the  Interconnection 
of  two  modules  A and  B In  the  following  way: 

errors  E^^  are  associated  with  module  A 
errors  Eg  are  associated  with  module  B 
errors  Ej^  are  associated  with  the  Interaction  of  A and  B. 

If  testing  and  usage  of  A end  B has  bean  carried  out  Independently  using 
simulated  environments  then  an  estlzwte  of  E^^  will  be  available  and* 
assuming  that  B Is  the  older  (data  base)  module  or  system  to  which  A Is 
added,  some  estimate  of  Eg  will  also  have  bean  made.  When  the  new  software 
A Is  connected  to  the  old  system  B two  new  sources  of  error  will  appear. 
Firstly,  there  will  be  the  errors  ®AB  which  are  the  result  of  interface 
mismatch.  Secondly,  there  la  the  fact  that  as  the  new  module  provides 
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soma  aaw  function  to  systan  B tha  uaaga  of  that  systam  and  hence  Its 
rallablllty  must  change.  Conceptually  this  amounts  to  discovering 
errors  which  were  In  but  ware  not  discovered  because  of  previous 
use  of  the  system.  Although  a simulation  of  the  envlromtent  of  B could 
reasonably  hope  to  find  most  of  the  errors  In  A»  It  seems  tmreasonable 
to  hope  that  environment  simulation  would  detect  many  of  the  other 
errors  which  will  appear  when  the  two  components  are  Integrated. 

Bealdes  the  Initial  difficulty  of  predicting  what  will  happen 
lAen  modules  are  integrated*  It  Is  extremely  difficult  even  after  module 
A has  gone  live  and  been  added  to  B to  give  any  precise  meaning  to  the 
reliability  of  A.  System  errors  will  presximably  occur  and  can  be 
detected.  But  analogous  to  the  classification  of  operating  system 
failures  Into  hardware*  software  and  operator  generated  there  are  now 
some  very  arbitrary  decisions  to  be  made  as  to  whether  the  failure  was 
caused  by  nodule  A or  not.  Corrective  action  will  often  Involve  both 
modules.  The  requlresients  (used  to  define  failures  In  the  first  place) 
of  one  or  both  modules  may  also  have  to  be  adjusted. 

What  emerges  from  this  discussion  Is  that  It  Is  extremely 
treacherous  to  try  to  define  the  reliability  of  a program  which  has  to 
operate  within  a given  software  environment.  In  such  cases  the  failures 
whose  frequency  Is  being  modelled  are  not  well  defined  events.  The  bug> 
counting  approach  Is  best  suited  to  describing  the  behaviour  of  an 
Isolated  total  system.  For  a total  software  systam  which  Is  nsneged  as 
a collection  of  components*  the  error-counting  approach  appears  to  give 
too  little  Information  about  the  locality  of  errors  and  the  effects  of 
Interfaces  to  be  useful. 
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Since  there  Is  no  meaningful  probabilistic  description  for  the 
failure  rate  of  small  programs  whose  Input  space  and  functional  reqtxlre- 
ments  can  be  completely  specified*  software  reliability  theory  Is  confined 
to  describing  the  behaviour  of  larg».acale  systems.  These  large  programs 
tend  to  be  Implemented*  then  developed  and  maintained  over  a considerable 
period  of  time*  During  this  time  their  functional  requirements  will  almost 
certainly  be  refined  and  may  be  significantly  altered  to  meet  new  usage 
desiands  which  were  not  foreseen  by  the  original  designers.  Several  authors 
have  observed  that  these  tendencies  are  quite  general  and  can  be  formalized 
» laws  which  appear  to  govern  the  growth  patterns  of  a wide  class  of  soft- 
ware systesis  (2)  (6).  The  complete  history  of  a software  project  provides 
globel  (In  time)  Information  on  program  quality  irtileh  Is  complementary  to 
the  view  associated  with  error  counting.  Also  the  fact  that  software 
specifications  do  change  over  time  presents  some  additional  problems  In 
failure  Interval  analysis.  It  would  therefore  seem  that  some  synthesis 
of  the  life  cycle  and  reliability  measurement  approaches  would  be  fruitful. 

All  published  error Tountlng  predictions  of  reliability  have 
been  smoothing  models.  That  Is  to  say  that  future  Instantaneous  failure 
rates  are  estimated  by  averaging  past  failure  rates*  giving  greatest  weight 
to  the  moBt  recent  observations  and  extrapolating  all  data  points  along  an 
exponential  curve  In  time.  This  method  Is  extremely  conservative  In  the 
sense  that  It  makes  no  provision  for  building  In  Information  about  the 
development  planned  for  the  software.  Hence  the  method  Is  best  suited  to 
very  short-term  forecasting  where  the  main  effect  Is  removal  of  errors. 


For  short  Intervals  extrapolation  of  recent  trends  gives  more  definite 
projections  than  the  plans  for  project  development  whose  effect  Is  botmd 
to  be  difficult  to  qiiantlfy. 


A common  feature  of  very  large  or  complex  software  Is  that  It 
has  to  be  developed  Incrementally.  In  other  words,  phases  of  testing  and 
debugging  alternate  with  the  Inclusion  of  new  code  expanding  the  capa- 
bilities of  the  sirstem.  This  Is  done  partly  to  ensure  that  the  task  of 
Integration  Is  carried  out  with  as  little  of  the  system  as  possible  at 
each  stage  and  partly  because  some  of  the  functional  requirements  of  the 
system  become  apparent  only  during  the  development  process. 

The  fact  that  systems  for  which  reliability  modelling  Is  meaning- 
ful are  usually  developed  In  this  way  raises  the  question  of  how  often 
estimates  of  the  number  of  errors  In  a program  should  be  reinitialized. 

In  essence  the  proposed  fitting  of  an  exponential  curve  to  the  instantaneous 
failure  rate  Is  mathematically  equivalent  to  extrapolating  backwards  from 
each  observed  failure  and  avera<3lng  the  results  to  obtain  an  Increasingly 
accurate  estimate  of  the  Initial  number  of  errors  In  the  program  at  time 
tg  say.  When  the  program  enters  a new  phase  of  development  say  at  time  t]^ 
new  code  will  be  added  and  presumably  new  errors  Injected  with  It.  Sub- 
sequent failure  data  If  extrapolated  back  to  time  t^  gives  an  estimate  not 
of  the  number  of  errors  in  the  system  at  time  tg  but  rather  an  estimate  of 
that  number  of  errors  which  If  they  had  been  present  at  tg  and  all  the 
system  had  been  available  for  testing  In  the  Interval  (tg,  ti)  would  have 
left  the  actual  number  of  errors  In  the  enlarged  system  at  time  t^.  Thus 
after  Injection  of  new  code  Into  the  system,  failure  data  from  before  that 
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tlM  biases  the  reliability  measurement  since  it  is  effectively 
estimating  a different  quantity.  It  so  happens  that  the  exponential 
law  for  error  elimination  has  the  effect  of  attaching  a relatively 
heavy  weight  to  the  most  recent  observations  and*  therefore*  prevents 
this  bias  from  becoming  overwhelming  In  the  long  term.  However*  It 
should  be  possible  to  obtain  more  accurate  estimates  by  cutting  out  old 
data  and  reinitialising  the  model. 

The  above  paragraph  explains  the  importance  of  keeping  track 
of  developments  to  the  system  and  using  them  in  the  reliability  analysis. 
Up  to  this  point  reliability  has  been  assumed  quite  specifically  as  msan 
failure  rate.  There  Is  another  aspect  of  software  system  quality  which 
describes  its  ability  to  cope  with  unpredictable  shocks  - namely  Its 
ability  to  be  developed  and  extended  in  ways  unforeseen  by  the  original 
designers.  This  quality  Is  sometimes  referred  to  as  maintainability. 

Uhen  a software  system  is  developed  Incrementally*  Its  reliability  will 
develop  in  sawtooth  pattern  with  sharp  rises  In  failure  rate  corresponding 
to  the  Inclusion  of  new  code  and  gradual  declines  In  failure  rate  cor> 
responding  to  the  working  out  of  errors  during  phases  of  testing  and 
debugging.  The  life  cycle  approach  to  software  would  try  to  Identify 
changes  in  the  mslntalnablllty  of  the  system  by  noting  changes  in  the 
shape  of  each  repeated  section  of  the  reliability  curve. 

Life  cycle  phenomena  could  in  turn  be  used  to  refine  the 
reliability  model.  For  example  the  slowing  of  the  rate  of  growth  of 
software  systems  Is  usually  attributed  to  their  becoming  less  well 
structured  es  a result  of  taking  on  functions  which  were  not  part  of  the 
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original  design.  Loss  of  structure  In  a prograa  makes  errors  markedly 
difficult  to  diagnose.  One  therefore  expects  the  ratio  of  observed 
failures  to  errors  removed  to  gradually  Increase  as  the  system  ages. 

Another  suggestion  from  life  cycle  analysis  Is  that  errors 
may  propagate  In  an  old  poorly  structured  system.  Correction  of  one 
error  may  create  others.  In  (1)  the  chain  reaction  possibilities  In 
this  picture  were  Invoked  to  explain  dramatic  changes  la  reliability 
between  consecutive  releases  of  an  operating  system.  This  possibility 
has  obvious  ramifications  for  error  counting  models. 

Conclusion 

Bug -counting  models  of  reliability  provide  some  useful  manage- 
ment Information  when  used  to  describe  the  observed  failure  rate  of  an 
operating  system  working  In  one  type  of  Installation  with  a fairly  static 
environment.  To  refine  these  models  and  wld^n  their  range  of  applicability 
requires: 

(1)  Identification  of  those  qualities  of  the  usage  environment 
idxlch  can  be  altered  and  affect  observed  failure  rates; 

(11)  better  understanding  of  how  to  handle  the  reliability  of 
Interacting  software  components  and  Interfaces;  and 
(ill)  the  synthesis  of  life  cycle  and  failure  rate  views  In 

order  to  obtain  explanation  of  how  software  reliability 
changes  In  the  longer  term. 
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l . Introduction 

M>’  intention  in  writing  this  paper  is  to  provoke  discussion  about 
two  aspects  of  software  reliability  m.easurenent . The  method  I shall 
•idopt  is  one  of  critical  analysis  of  some  previous  research,  together 
.-.ith  tentative  suggestions  for  future  directions.  In  order  not  to 
overburden  the  argument,  I shall  not  bend  over  backivards  to  praise  the 
positive  aspects  of  work  I criticise  : I am  sure  the  authors  concerned  , 
will  be  quite  capable  of  performing  this  task,  and  in  so  doing  help  make 
f for  interesting  discussionl 

The  first  part  of  the  paper  will  be  concerned  with  the  basic  problans 
of  definition.  Before  tlais  provokes  a cry  of  despair  from  the  reader,  I 
should  enqjliasise  that  I see  my  task  as  being  as  much  to  convince  him  of 
the  necessity  of  the  operation,  as  selling  my  own  favoured  solutions.  >J>' 
general  thesis  will  be  that  many  of  the  reliability  measures  adopted  for 
software  rely  far  more  upon  hardware  analogies,  often  unconsciously  adopted, 
than  on  a careful  analysis  of  the  special  requirements  of  software.  Such 
an  analysis,  I believe,  could  reveal  much  research  to  be  a mixture  of 
technically  sophisticated  falsehood  and  meaningless  nonsense.  This,  if  true, 
is  bad  enough  : but  worse  happens  when  these  ideas  are  exposed  to  the 
innocent  conputer  scientist  having  no  background  in  reliability  theory.  I 
shall  quote  from  a recent  paper  in  Computer  [13] , surveying  methods  of 

ac. hieving  reliable  software,  whose  authors  manage  to  be  confusing  about  even 
the  old  hardware -derived  reliability  concepts,  before  they  move  on  to  ti.e 
safer,  deterministic  ground  of  proving,  structuring,  testing,  etc. , which  is 
their  forte. 


j 

i 
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Even  if  satisfactory'  answers  can  be  achieved  to  the  problems  in  this 
area,  it  needs  to  -be  pointed  out  that  we  shall  only  be  in  a position  (using 
the  hardware  analogy)  to  treat  software  as  a black  box  (component) . Let  us 
not  forget  that  the  huge  achievement  of  hardware  reliability  theory  was  the 
provision  of  methods  of  incorporating  both  stochastic  information  about 
component  failure  behavaour  and  deterministic  iniorraation  of  the  system's 
structure  of  components.  Something  similar  is  needed  for  software, 
particularly  in  view  of  t.he  modem  approac.hes  of  modularity  and  structured 
prograiming.  M>'  own  belief,  however,  is  tliat  the  hardware  analog>'  will  agam 
be  of  little  direct  assistance.  The  second  part  of  the  paper  will  look  at 
this  question,  and  again  I hope  to  be  able  to  suggest  areas  in  which  effort 
should  be  concentrated.  1 


Classical  reliability  measures 
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Let  us  begin  by  looking  briefly  at  the  measures  v.'hich  have  been  used 
in  the  liardware  field.  One  of  the  most  careful  accounts  is  still  tliat  of 
liar low  and  Proschan  [1] . They  give  two  basic  measures  which  will  generally 
have  meaning  : reliability  and  availability. 


2.1  Ileliability 


Barlow  and  Proschan  give  two  definitions, 
only  the  more  general  one  : 


iVe  shall  consider  her*. 


i 
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Interval  reliability  is  the  probability  that  at  a specified  tune,  T, 
the  system  is  operating  and  will  continue  to  operate  for  an  interval 
of  duration  x. 


We  shall  denote  interval  reliability  by  RCx,T) . Of  course,  in  many  cases 
we  shall  be  interested  in  limiting  interval  reliability, 

say  R(x),  given  by 


RCx)  « lim  R(x,T)  , 

T->- 

assuming  .that  this  limit  does  exist. 

2.2  Availability 

There  are  two  definitions,  both  being  in  common  use: 

Pointwise  availability  is  the  probability  that  the  system  will  be  able  * 
to  operate  within  the  tolerances  at  a given  instant  of  time,  t. 

We  shall  denote  it  by  G Ct) . 

Interval  availability  is  the  expected  fraction  of  a given  interval  of 
time,  (.a,bj,  t.hat  the  system  will  be  able  to  operate  within  the  tolerance? ^ 
(repair  and/or  replacement  allowed) . ' \ 

We  shall  denote  this  by  H(a,b) . Barlow  and  Proschan  define  limiting  inten’-al  | 
availability  to  be  1 


lim  H(0,T)  . 

: 1 

= j 

It  is  sometimes  sensible  to  consider  also  a limiting  version  of  G(t)  as  t-^*  , | i 

we  shall  call  this  limiting  pointivise  availability. 

2.3  How  adequate  are  these? 

Given  that,  for  reasons  of  simplicity,  it  is  necessary  to  sinumarise 
all  available  information  into  a single  numerical  measure  of  reliability, 
then  one  or  other  of  these  definitions  will  be  appropriate  for  most  situations. 

My  own  opinion,  however,  is  that  there  is  usually  no  such  overriding  need  for 
simplicity  and  that  more  insight  into  the  process  of  failures  would  be  gained 
by  considering,  say  percentiles  of  time-to-next-failure  distributions. 


Of  course,  striking  a balance  between  siTiplicity  and  infomati/eness  is 
a matter  for  fine  judgment,  but  it  does  seem  tiiat  in  much  writing  on 
hardware  reliability  there  is  a preference  for  the  simplistJc  rather 
thaji  the  simple.  I am  thinking,  in  particular,  of  the  enormous 
popularity  of  mean-time-to-next-failure,  mean-time-between-foilures,  and 
similar  measures.  Even  in  the  case  of  hardv/are,  it  is  regrettable  that 
large  amounts  of  failure  data  should  still  be  stmtnarised  in  S’jch  a crude 
fashion;  for  software,  however,  I hope  to  show  later  in  this  paper  that 
use  of  stich  measures  can  be  very  misleading.  ^ 

I have  a technical  criticism  of  one  of  tlic  availability  definitions 
vdiich,  again,  ulll  be  particularly  serious  in  tlie  case  of  software.  j 

Consider  interv'al  availability  : "the  expected  fraction  of  a given  interval  *- 

of  time  that  the  system  will operate ".  V^hat  is  really  of  interest 

is  the  actual  fraction  of  time  the  system  will  operate,  but  of  course, 
this  is  a random  variable.  Merely  quoting  the  e.xpected  value  of  the  random 
variable  gives  no  idea  of  iiow  much  tl\e  actual  result  might  deviate  from 
this.  We  need  the  distribution  of  the  fraction  in  order  to  be  able  to 
calculate  a confidence  mten'al. 


In  many  practical  situations,  of  course,  it  is  the  limiting  form  of 
the  availability  definition  which  is  most  appropriate.  In  such  cases  it 
might  be  thought  that  the  fraction  of  an  interval  (0,T)  that  the  system 
will  operate  would  converge  in  probability,  for  large  T,  to  the  limiting 
expected  fraction  Cliiniting  availability).  This  result  would  normally  be  i 
established  using  Tdiebycheft's  inequality  ([3],  p.i6).  Unfortxinately,  * 
convergence  cannot  be  guaranteed,  fn  fact,  if  we  model  the  system's 
behaviour  by  an  alternating  renewal  process  ([■+],  p.  80  et  seq.)  with  the 
two  types  of  time  intennl  representing  operating  and  repairing,  then  it 
can  be  shown  t.hat  the  fraction  of  time  spent  working  does  converge  in 
probability  to 


(where  u and  v are  the  merns  of  the  time-to-failure  and  time-to-repair 
w r 

distributions)  as  long  as  these  means  exist.  If  these  distributions  do  not 
have  moments,  convergence  may  still  taxe  place,  but  it  is  possible  to 
construct  examples  where  it  does  not.  If  such  a situation  should  be 
encountered,  where  the  limiting  interval  reliability  does  not  have  the 
interpretation  of  (probabilistic)  limiting  fraction  of  time  operating,  it 
seems  to  me  to  be  difficult  to  assign  it  any  practical  meaning.  It  is  my 
contention  that  we  are  more  likely  to  encounter  this  kind  of  difficulty 
with  software  than  with  hardware. 

All  of  the  above  comments  and  criticisms  have  been  essentially  of  the 
same  nature  : we  should  be  extremely  careful  of  replacing  the  wealth  of  ‘ 
information  in  a probability  distribution  ;nth  simple  summaries  - ;diether 
these  be  parameters  or  moments  of  the  distributions.  The  extraordinary 
pre-eminence  of  the  exponential  distribution  in  hardxvare  I'eliability  is  li 
probably  the  b.istorical  reason  for  the  obsession  with  the  mean.  At  the 
very  least  it  e.xplains  the  confusion  between  mean  time  between  failures 
and  mean  time  to  next  / -'  iure  (i.e.  measured  from  an  arbitran'  time  origin^j . 
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I not  a failure) , since  it  is  only  for  the  exponential  disti ihution  that 

these  are  identical.  There  are,  hov.ever,  good  reasons  for  thinking 
’ that  certain  complex  Iiardware  devices  might  follow  an  expon-'ntial  law 

([1],  pp.  13-22).  These  reasons,  unfortu^tely.  will  not  justify  its 
application  to  software  . It  remains  an*  open  question  whetl-er  a 
failure  la\v  can  be  developed  ^diich  reflects  the  structure  of  software, 

; but  in  the  absence  of  such  a law  we'should  beware  of  applying  to 

[ software  those  hardware  reliabj.lity  concepts  which  have  the  exponential 

distribution  as  their  basis 

i 
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3.  Softv.'are  reliabilitv  msasuiement 


In  the  previous  section  we  have  seen  instances  where  the  hardware 
reliability  experience  iias  prc/ed  a mixed  blessing  for  software 
reliability  measurement.  Even  in  situations  where  the  hardware  concepts 
might  be  appi*opriate,  there  is  a great  deal  of  confusion  about  these 
most  basic  definitions.  This  is  exemplified  by  the  following  passage 
from  an  otherwise  informative  recent  article  [is]  : 

"Three  tewsS  are  cormonly  used  to  measure  reliability  of  a system: 
'availability' , 'reliability'  i^nd  'mean  time  to  repair  (or  recover)'. 
Availability  is  defined  as  the  probability  that  the  system  is  capable  of 
being  used  at  time  T.  Reliability  is  defined  as  the  probability  that  the 
time  between  component  failures  is  greater  than  t.  Thus  (sic) , mean  time 
between  failures  is  the  usual  measure  of  reliability. 

"Availability  is  maximised  by  maximising  the  mean  time  between 
failures  and  minimising  the  mean  time  to  repair  or  recovery 

In  what  follows  I shall  try  to  give  a personal  view  of  some  aspects 
of  softivare  reliability  measure.ment . The  approach  may  appear  more 
critical  and  descriptive  tlian  prescriptive,  it  being  easier  to  point  to 
defects  of  existing  tecliniques  than  suggest  new  ones.  In  any  case  it 
seems  to  me  that  the  role  of  a probabilist  is  to  describe  carefully  the  1 
implications  of  certain  mathematical  and  probabilistic  ideas  and  allow 
the  customer  to  choose  the  ones  which  are  most  appropriate  for  his 
applications . 

3.1  Bugs  bared 

At  some  risk  of  oversimplifying,  bugs  (or  errors)  can  be  defined 
as  those  'defects'  in  the  program  wtiich  cause  failures  in  its  dynamic 
operation,  .is  sixh,  they  are  the  things  which  the  software  engineer  will 
try  to  eliminate  during  pregr^.  testing  and  development.  Mills  [13,14] 
would  prefer  to  concentrate  effort  on  ensuring  they  never  get  into  the 
program  in  the  first  place. 

I am  not  convinced  that  the  advocates  of  measuring  reliability 
via  bug  counting  have  presented  a strong  case.  It  seems  to  me  that  our 
objective  should  be  to  laeasure  the  quality  of  the  "behaviour"  of  the 
software,  its  operational  reliability,  rather  tlian  the  quality  of  its 
"state".  A "goexi"  progrXm  is  defined  in  terms  of  what  it  does,  not  what 
it  is.  Tile  proof  of  the  pudding 

Of  course,  when  the  program  fails  in  some  way,  it  is  the 
software  engineer's  job  to  find  the  cause (s)  of  that  failure  : he  has  to 
eliminate  bugs.  Although  the  bug  eliminator  and  the  reliability  measurer 
may  be  embodied  in  the  same  person,  the  operations  are  quite  different  and 
only  confusion  ensues  from  identifying  them.  In  fact  I suspect  that  the 
distinction  might  have  important  economic  and  social  consequences  : would  I 
it  not  be  a great  help  in  getting  better  qualit>’  software  to  insist  that  ‘ 

the  reliability  improvement /measurement  interface  coincide  with  that  of 
contractor/ customer?  I have  no  direct  e.xperience  of  US  practice,  but  I 
know  of  cases  in  the  UK  where  contractors  have  acted  as  their  own  judge  and 
jury. 
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It  can  be  argued  tliai  the  state  of  a program  (number  of  bugs) 
determines  its  performance  (operational  reliability),  but  any  relationship 
here  is  likely  to  be  very  com-J  icated  and  unknov.n.  Certainly  the  kind  of* 
assunptions  which  have  been  made  seem  very  naive,  thus  Sliooman  [18]  says: 

"...the  software  failure  rate  (crash  rate)  is  proportional  to 

the  number  of  remaining  errors." 

I have  never  seen  a program  uhere  this  assumption  could  be  valid.  It  is 
easy  to  imagine  a scenario  where  a program  with  two  bugs  in  little 
exercised  portions  of  code  is  more  reliable  than  a program  with  only  one, 
frequently  encountered,  bug. 

I concede,  though,  that  it  might  sometimes  be  possible  to  model 
realistically  the  relationship  between  operational  reliability  and  number 
of  residual  bugs.  But  why  bother?  V.hy  introduce  this  extra  risk  of 
modelling  error,  xvhen  we  can  do  anything  we  want  in  terms  of  operational 
reliability,  directly  ? Tlius,  for  example,  if  we  want  estimates  of  the 
debugging  time  needed  to  attain  a specified  reliability  (NB  specified  in 
terms  of  the  ^antity  of  real  interest,  operational  reliability),  this  can 
be  achieved  via  a suitable  model  based  upon  operational  reliability  (see, 
for  example,  [11]  p 113). 

3.2  How  to  measure  operational  reliability 

Having  convinced,  I hopev  that  operational  reliability  is  wliat  we 
should  be  measuring,  we  now  have  to  consider  ways  of  doing  this.  Although 
the  problem  is  a dual  one  of  modelling  and  estimation,  my  present  concern 
is  ^vith  the  former,  since  it  is  in  the  modelling  stage  that  I believe  we 
must  be  concerned  with  some  quite  unique  properties  of  software. 

We  must  consider  a renewal -type  process  in  continuous  time,  the 
successive  renewals  representing  successive  failures  of  the  software.  For 
simplicity  assume  initially  that  "repairs"  are  instantaneous  (there  is 
surprisingly  little  information  available  about  repair  times).  It  is 
probably  worth  mentioning,  also,  that  "time"  should  be  "execution  time" 

(see  Musa  [16])  : clearly  there  can  be  no  failures  when  the  program  is  not 
being  used. 

Such  rene^val  processes  can  be  characterised  either  in  terms  of 
their  successive  inter-event  times,  or  via  the  numbers  of  events  in  fi.xed 
time  intervals.  The  former  method  is  the  more  appropriate  one  for  our 
purposes.  My  contention  iS  that  tlie  distributions  of  times  between  failures 
may  have  unusual  properties,  in  particular  their  moments  may  not  exist  (vis, 
are  infinite).  If  this  v;ere  true  then  classical  measures  such  as  mean-time- 
to-next-failure  (mttf),  mean-time-between-failures  (mtbf)  would  be  infinite, 
and  any  estimates  of  them  (altliough  finite  themselves)  would  be  meaningless. 

This  assertion  is. quite  revolutionary,  and  it  wuld  be  pleasant  to 
be  able  to  say  that  I have  evidence  to  support  it  from  actual  software 
failure  data.  Unfortunately  this  is  not  the  case.  As  far  as  I am  aware 
there  is  no  good  statistical  test  of  the  h>'pothesis  that  a set  of  data  comes 
from  a moment-less  population.  In  any  case,  the  nature  of  the  problem  is 
likely  to  require  that  such  a test  be  based  upon  a large  amount  of  data. 
Software  failure  data  is  still  notoriously  difficult  to  obtain  in  large 
quantities. 
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In  fact  it  is  quite  possible  that  belief  in  a program  get  progressively 
worse  if  failures  are  sufficiently  frequent,  a situation  which  may  occur 
in  practice  (see  Shcoman  and  Xatarajan  [20],  footnote,  p.l64].  The 
strength  of  tlie  model  lies  in  its  ability  to  produce  the  appropriate 
answer  automatically  : reliability  groxvth  or  reliability  decay  do  not 
need  to  be  input  a priori.  The  most  important  advantage  of  Ba)'esian 
models,  though,  lies  in  their  ability  to  reflect  that  attitude  to 
programming  which  has  Mills  as  its  best  known  proponent.  He  states  [13]: 
"....never  finding  the  first  error  gives  m.oi*e  confidence  than  finding 
the  last  error".  The  Bayesian,  "no  news  is  good  news"  property 
represents  this  exactly  : periods  of  failure-free  working  cause  the 
reliability  to  inprove.  It  is  interesting  to  compare  this  with  the  bug- 
counting methods  of  Jelinski  and  Moranda,  Shopman  etal:  here 
reliability  mprovemenz  can  only  take  place  at  a failure,  since  it  is 
only  at  such  a time  that  an  error  can  be  removed.  Tney  are  tlius  in 
direct  contradiction  to  the  Mills  doctrLne : increasing  confidence  Ln  the' 
program  as  failures  occur I 

3.4  Reliabilitv  versus  utilitv 

- - 

A surprising  omission  from  most  of  the  literature  of  software 
reliability  is  a concern  for  the  consequences  of  failures  in  terms  of 
economic  (or  other)  cost.  This,  again,  may  be  a result  of  the  subject's 
close  connections  with  liardware  reliability,  which  has  traditionally 
concentrated  on  modelling  the  failure  process.  The  omission  is  rectified 
to  some  e.xtent  in  the  wider  software  engineering  context  : management 
techniques,  on  the  whole,  being  cost-conscious.  This  literature, 
unfortunately,  tends  to  be  more  qualitative  tlian  quantitative. 

Hardware  reliability  tells  us  how  to  make  a system  as  reliable 
as  we  desire  by  using  sufficiently  many  conponents  of  a given 
unreliability.  We  can  thus  diminish  the  life-time  cost  of  particular 
t>T3es  of  failure  by  making  their  frequency  of  occurrence  sufficiently 
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3.3  Software  engineers  should  be  Bavesians 


In  this  section  I hope  to  convince  you  that  we  shculd  use 
Bayesian  interpretations  and  methods  for  software  reliability.  There 
are  at  least  two  good  reasons. 

In  the  first  place,  the  subjective  interpretation  of  probability, 
which  is  usually  associated  with  the  Bayesian  school,  seems  more 
appropriate  for  sofware  than  a frequent ist  approach.  Because  of  the  "one- 
off”  nature  of  soft\\'aro,  there  is  usually  no  '^ense  in  \vhich  one  can 
envisage  an  ensemble  of  repetitions  of  the  reliability  measuring  operation 
upon  which  a frequentist  interpretation  would  depend.  This  is,  of  course, 
an  ijiportant  difference  bet^veen  hardware  components,  which  are  essentially 
repeatable,  and  soft^vare. 

The  second  reason  is  more  pragmatic.  The  essence  of  Bayes 
Theorem  is  tliat  it  provides  a means  of  continuously  updating  previous 
reliability  measurements  in  the  light  of  new  data.  Consider,  as  an  ex^le, 
the  kind  of  calculation  uhich  is  possible  with  our  own  Bayesian  reliability 
growth  model  [10,12] . Figure  1 gives  a portion  of  the  plot  of  instantaneous 
failure  rate  which  can  be  obtained,  showing  the  development  as  time  passes. 
Thus,  prior  to  the 
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ith  failure,  whilst  the  program  is  working  without  fault,  confidence  in  it 
increases  and  the  failure  rate  falls.  IVhen  the  i th  failure  occurs, 
confidence  drops  and  the  failure  rate  increases,  but  immediately  falls  by  a 
finite  amount  owing  to  the  efficacy  of  the  repair.  .After  the  (instantaneous) 
repair  the  program  works  again  in  continuous  time,  and  during  this  period 
the  failure  rate  again  falls  continirausly.  If  our  belief  in  the  skill  of 
the  debugger  is  not  sufficiently  great  to  overcome  our  pessimism  at  the 
occurrence  of  a failure,  then  a plot  such  as  Fig.  2 will  result. 


Support  for  this  idea,  then, must  be  analytical  rather  than 
directly  evidential.  In  the  first  place  it  could  be  asked,  slightly 
frivolously,  what  evidence  there  is  for  the  e.xistence  of  mor..^nts. 

Those  people  who  wish  to  estimate  mttf  should  be  asKed  to  furiush 
evidence  that  the  quantity  they  are  estimating  does  indeed  ex:ist. 

One  can  always  obtain  a finite  average  of  some  data,  after  all,  but 
it  may  not  estimate  a population  mean.  Nkare  seriously,  there  is  the 
unique  property  of  soft^-.are  that  it  suffers  no  natural  degradation: 
once  perfect,  it  will  never  fail.  If  we  concede,  therefore,  that  there 
is  some  chance  Chowever  small)  that  the  progrrm  is  perfect,  ^en  the 
mean  time  to  failure  must  be  infinite  (see  [G]) . In  fact  this  case  is 
so  ectreme  that  not  even  fractional  iiioments  wo»U.d  exist.  But  is  it  such 
an  extreme  assumption  ? Vhen  we  come  to  look  ^t  the  problem  of 
incorporating  structural  information  into  our  reliability  modelling  we 
shall  consider  modular  programming  - it  does  not  seem  unreasonable  to 
believe  that  a small  enough  module  might  be  perfect.  In  fact  Mills  goes 
much  further,  £ sorting  that  top-down  structuring  is  likely  to  produce 
perfect  programs,  even  when  these  are  large  [14]  : 

"The  n»^  reality  is  that  you  can  learn  to  consistently  write 
programs  whirh  are  cotrecC  ab  initio,  and  prove  to  be  error  free  in  their 
debugging  and  sxabsequent  use." 

Finally,  it  is  possible  to  construct  models  \diich  have  moment-less  time- 
to-feilure  distributions  as  a consequmce  of  quite  unexceptionable 
assumptions  about  the  properties  of  sofovare.  Our  own  model  [10,12]  is 
one  such  case.  In  rhis  work  we  took  failure  tate  as  the  measure  of 
interest,  and  modelled  the  improvement  of  reliability  in  a subjective, 
Bavesian  fcf.  Che  error  removal  models  of  Shooraan,  et  al  [17,18, 

ZOJ , and  Jelinski  and  ^fora^a!a  [a]).  TEis  model  has  great  flexibility  and 
ai  Tnwg  exact  grributiong  o£  time-to— feilure  to  be  computed,  with 
associated  percentiles,  medians,  etc.,  in  addition  to  faiJLure  rate 
nieasiires.  If  we  accept  the  plausibility  of  models  of  this  kind,  it  setss 
to  me  that  we  should  not  baulk  at  any  consequences,  such  as  infinite  mean 
times  to  failure. 

If  we  are  to  esche;'  the  use  of  mean  time  to  failure,  what 
measures  are  left?  I liave  already  argued  that  we  should  be  altogether 
less  obsessed  with  single  measures  : preferring  instead  distributions 
from  ^duch  we  can  calculate,  if  requi^,  many  appropriate  measures. 

Thus  front  a to  failure  distribution  we  could  quote  the  probability 
that  the  rime  to  next  failure  exceeded  aiv  reipired  value.  This  enables 
us  to  tailor  our  measures  to  the  particHar  situation  in  hand,  single 

measure  of  quality  is  really  required,  failure  rate  (l^ard  rate)  is  tnore 
plausible  than  mean  time  to  failure,  since  it  will  exist  under  much  wider 
conditions.  Indeed,  for  the  decreasing  failure  rate  [1]  situations  which 
we  hope  to  encounter  in  software,  instantaneous  failure  rate  has  a more 
natural  intuitive  interpretation  than  [instantaneous)  mean  tune  to  failure. 

Notice  that  these  considerations  about  the  possible  nOT^^tence 
of  the  mean  time  to  failure  may  cause  difficulties  with  the  definition  of 
availability,  as  mentioned  earlier.  Even  if  the  mean  does  not  exist,  it 
may  be  that  the  fraction  of  time  available  converges  in  probability  to  some 
COTStant,  but  this  cartret  be  assumed.  In  fact  de^il^  knowledge  of  the 
distributions  of  ti::.3  <. , faiiur?  ard  time  to  repair  will  be  needed.  Scant 
attention  seem  to  iiave  oeen  paid  to  repair  time  distributions  in  the 
literature. 
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small.  Since  we  do  not  yet  really  have  such  techniques  available  for 
software  (and  we  may  never  get  them) , it  seems  to  me  particularly 
inportant  that  we  at  least  esi  imate  life-time  costs  of  our  systems. 

In  fact,  I wonder  how  many  projects  we  would  embark  i^on  if  we  knew 
how  much  th<»y  would  cost  over  their  life-time,  rather  than  merely  to 
create 


It  does  not  matter  much  ivhether  we  decide  to  adopt  utility 
(profit)  or  cost  as  our  criterion  : a choice  v/ill  usually  be  dictated 
by  personalit)'  and  circumstances.  Tlius  Buzen  et  al  [2]  coiroared  the 
utilities  of  virtual  machine  and  conventional  operating  systems  : in 
fact  they  assumed  constant,  linear  utilities.  In  in>'  om  work  I have, 
pessimistically,  used  costs  [9]  : the  system  can  fail  in  different 
modes,  each  particular  failure  resulting  in  a (random  variable)  cost 
with  a distribution  typical  of  the  mode.  Interest  then  centres  upon 
the  total  cost  of  xtinning  the  system  for  time  T.  Clearly  this  is  a 
random  variable,  and,  in  order  to  be  true  to  my  evangelism  of  earlier 
sections,  it  would  be  most  informative  to  obtain  its  distribution,  in 
particular  as  T This  can,  in  fact,  be  done.  In  practice,  of 

course,  the  system  will  "earn"  utility  during  its  working  periods  and 
"lose"  during  failed  periods  (cost  of  repair,  loss  of  revenue,  etc.). 
This  refinemeiit  does  not  seem  difficult  to  achieve  in  both  the  above 
models. 


It  is  often  argued,  against  this  kind  of  approach,  that  we 
hardly  ever  have  much  information  about  costs.  This  seems  unnecessarily 
defeatist  : after  all,  merely  count in j?  failures  is  equivalent  to  giving  a 
unit  cost  to  each.  Surely  we  always  Imow  better  than  that? 


4. 


Structural  models 


Most  of  the  earlier  part  of  this  paper  was  concerned  ''ith  measuring 
the  quality  of  a single, "black-box"  program  ; the  equivalent  of  a device 
or  component  in  the  hardware  terminology.  Fortunately,  we  normally  have 
a large  amoitnt  of  information  available  about  the  structure  of  the  program, 
and  it  would  clearly  be  sensible  to  use  this  in  our  reiiaDillty  modelling. 
This  does  not,  however,  seem  to  be  an  easy  task.  Most  attempts  so  far  have 
looked  at  fairly  specialised  structure,  and  I »hall  briefly  review  two  of 
these  in  wliat  follows,  but  the  real  goal  is  the  modelling  of  more  general 
systems . 

One  of  the  most  interesting  attempts  to  go  beyond  the  black-box 
approach  is  that  of  Buten,  et  al  , [2]  who  studied  virtual  machine  (VM) 
techniques.  They  define  a VM  as  : 

a hardwnre-softw'are  duplicate  of  a real  e.xisting  con^juter 
system  in  which  a statistically  dominant  subset  of  the  virtual 
processor's  instructions  execute  directly  on  the  host  processor 
in  native  mode. 

"Thus  a VM  provides  for  the  efficient  operation  of  one  or  more 
copies  of  a complete  computer  system  whicli  is  similar  to  the 
host  (or  real)  system." 

The  program  which  mediates  between  the  virtual  machine  and  the  actual 
resources  of  t.he  system  (host  machine)  is  called  the  virtual  machi.ne  monitor 
(V^^I) . This  provides  an  extremely  high  degree  of  isolation  tor  each  vM 
running  under  its  control,  so  that  a failure  in  one  VA  will  not  affect  the 
operation  of  other  VM's.  Although  a VNW  failure  will  interfere  with  the 
operation  of  all  I’Ms,  there  is  good  reason  to  believe  that  the  can  be 
made  very  reliable.  The  example  used  in  [2]  is  a system  vdiich  supports  N' 
levels  of  multiprogramming.  The  usual  method  is  to  construct  a multi- 
programmed  operating  s>'stem  (OS-N)  which  runs  on  a bare  machine  and  support." 

N independent  user  profprams.  The  virtual  machine  approach  is  to  run 
copies  of  a moncprogrr--"ed  operating  system  (OS-1)  under  control  of  a Vl-M. 

By  assuming  all  recovt.-iv  tlmis  and  times  between  failures  to  be 
e.xpcnentially  distributed,  the  authors  are  able  to  prove  that  the  expected 
utility  of  V.-M/OS-l  is  greater  thin  or  equal  to  the  expected  utility  of 
OS-N,  with  quite  weak  assumptions  on  t.he  means  of  the  e.xponential 
distributions.  In  fact  I have  been  able  to  generalise  their  result  to  the 
case  where  failure  and  recovery  times  are  arbitrarily  distributed,  with 
means  (unpublished  work) . This  generalisation  makes  their  result  very 
powerful  and  is  a compelling  argument  for  the  VM  approach  in  such  situation.  . 

My  own  work  on  stinctural  models  [7,8,9]  has  been  Markovian  in  nature. 
The  assumption  here  is  that  the  program  can  be  decomposed  into  R subprogram* 
(modules)  and  execution  proceeds  by  suvitching  among  these  according  to  a 
Markov  (or  semi -Markov)  process.  Each  module  is  assumed  to  fail  in  a 
Poisson  process  (i.e.  e.xponential  inter-failure  times),  with  different 
moduiles  having  different  failure  rates.  A record  of  the  overall  failure 
process  is  thus  a sequence  of  failures  attributable  to  the  different  modules, 
and  its  probabilistic  description  is  exceptionally  con^jlex.  Fortuunately, 
simple  limiting  results  can  be  obtained  when,  as  seems  plausible,  the  failu  ; 
rates  are  ver>'  much  s'.,  .rr  thin  the  switching  rates  betuveen  modules.  It  ca.. 
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be  shoivn,  for  example,  that  the  overall  failure  process  (i.e.  merely 
counting  failures  and  not  attributing  tahem  to  a source  modula)  is 
Poisson,  with  an  easily  calculated  rate.  The  model  can  be  extended 
to  allow  each  failure  to  have  a (random  variable)  cost , pan icular 
to  its  module,  and  the  limiting  distribution  of  the  total  cost  in 
running  the  proga*am  for  a time  T can  be  obtained.  The  parameters  for 
these  limiting  distributions  are  quite  easy  to  estimate.  Foi  example, 
by  flagging  the  program  it  should  be  possible  to  obtain  good 
estimates  of  the  transition  probabilities  of  the  embedded  Markov  chain 
very  quickly.  Thus  the  model  could  genuinely  be  used  as  a design  tool: 
enabling  the  system  reliability  to  be  calculate-!  from  the  module 
reliabilities  (obtained  from  testing  these  ind  J.vidually)  together  with 
the  knowledge  of  how  tliey  interact  dynamically.  Of  course,  models  like 
this  are  intended  to  be  quite  general  and  flexible,  so  a measure  of 
their  quality  will  always  be  the  appropriateness  of  their  underlying 
assia^tions.  The  two  basic  assvanptions  here  are  that  the  s^ntching  be 
(semi-)  Markov,  and  each  failure  process  Poisson.  The  second  is  quite 
plausible  and,  at  least  for  the  limiting  theor>',  could  possibly  be 
relaxed.  It  is  in  the  limited  memory  of  the  Markov'’  assunption  that 
problems  are  likely  to  arise.  It  may  be  possible  to  test  this  in  some 
cases,  altlvough  this  needs  the  system  to  be  ninning,  Avhich  may  defeat 
the  purpose  of  the  exercise.  On  the  other  hand,  it  seems  likely  that 
we  could  sometimes  make  this  assumption  a priori,  for  example  in  real- 
time situations  where  the  program  is  reacting  to  a random  data  input 
stream.  Even  if  the  process  is  not  exactly  Markovman,  all  is  not 
necessarily  lost  : it  may  be  possible  to  make  it  Markovian,  albeit  at 
the  expense  of  a greatly  enlarged  state  space. 

Still,  the  chief  criticism  of  this  model  is  that  its  assumptions 
come  from  a desire  for  mathematical  tractability  rather  than  software 
plausibility.  Unfortunately,  the  kind  of  structure  which  is  natural  to 
software  tends  to  be  of  a static  kind  : the  program  as-it-is.  Our 
requirement  for  operational  reliability  demands  that  we  define  structure 
in  a dynamic  way  : the  progr’m  as -it -performs.  The  ideas  of  top-down 
structured  programming  might  be  able  to  bridge  this  gap,  but  the  way 
forward  is  not  clear  ;t  yiesent. 

However,  in  spite  of  all  this,  it  is  beginning  to  be  obvious  that 
techniques  such  as  modularity,  top-^wn  structuring,  etc.,  are  gaining 
in  popularity  for  good  practical  reasons  : they  appear  to  deliver  the 
goo^.  If  they  do  become  more  widely  utilised,  the  folloxving 
intriguing  question  becomes  of  interest  : do  these  methodologies 
inevitably  result  in  the  final  program  having  a particular  failure  lav? 
The  case  of  top-down  structuring  is  particularly  interesting  in  view  of 
the  simplicity  of  tlie  structure  - three  basic  program  figures  [13] . 
Alternatively,  in  modular  terminology,  where  we  imagine  a program  to  be 
conprised  of  subprograms  which  in  turn  are  con^rised  of  subsubprograms, 
etc.,  what  is  the  relationship  between  the  reliability  of  a module  and 
the  reliabilities  of  the  submodules  it  comprises?  It  seems  likely  that 
modules  and  submodules  have  the  same  failure  law  of  necessity  (w'ith, 
presumably,  different  parameter  values),  since  the  methodologv*  ’used  in 
their  creation  would  be  similar.  Uhat  would  such  a failure  law  be?  An 
answer  in  a special  case  is  giv'en  in  my  Markov  model,  where  it  is  proved 
that  if  the  subprogra.*:..'  -o’.Iow  a Poisson  law,  then  the  progi*am  itself 
will  (in  the  liiiit; . 1::  v -uli  la  of  great  interest  to  see  whether  there 

are  other  cases  where  the  form  of  the  failure  law  is  dictated  by  program 
structure . 
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Suirarur.'  and  recommendations 


(i)  Do  not  appl)’’  hardware  techniques  to  sofn-.'are  without  thinking 
carefully.  Sotware  oirters  trom  lurdiv-are  xn  in^jortant  respects  : we 
■Vgnore  these  at  our  peril.  In  particular 

(ii)  do  not  use  mttf,  mtbf  for  software,  unless  certain  that  they 
exist.  Even  then,  remsmber  tiiat 

(iii)  distributions  are  always  more  informative  than  moments  or 
parameters,  so  try  to  a\'Qid.  coiismment  to  a single  measure  of  reliability. 

Civ)  Sofpx'are  reliability  means  operational  reliability.  Mio  cares 
how  nany  bi^  are  m a program?  We  afe  concemea  witii  tneir  etfect 
its  operation. 

Cv)  Use  a Bayesian  approach  and  do  not  be  afraid  to  be  subjective. 
On  the  contrary,  all  our  stat^n^  win  ultimately  oe  aoout  our  beliefs 
in  the  qualit>'  of  programs. 

(vi)  Do  not  stop  at  a reliability  analysis  : try  to  model  the  life- 
time utility  (.or  cost)  of  programs. 

(vii)  Not^  is  the  time  to  devote  effort  to  structural  models. 

(viii)  Structure  should  be  of  a kind  appropriate  to  software, e.g.  the 
top-down  structuring  ot  Mills  . 

Finally » a-  plea  for  more  validation  of  models.  (Ve  can  argue  the 
relative  merits  of  our  particular  pet  schemes  for  ever^  but  they  must 
ultimately  be  tested  in  the  real  world.  IVbuld  anyone  like  to  support  a 
competition  between  rival  models  ^ providing  suitable  data  to  be  analysed  by 
the  different  methods?  ^^auld.  anyone  like  to  enter  such  a conpetition  .. ..? 
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A SUBJECTIVE  EVALUATION  OF  SELECTED  PROGRAM  DEVELOPMENT  TOOLS 


^ , ABSTRACT 

A program  davalopmant  project  can  be  viewed  as  a complex 
system  of  people,  of  hardware  and  software,  of  tools. 

Any  one  elraent  should  be  viewed  in  its  system  context. 
With  this  in  mind  a selection  of  analysis,  design, 
development,  and  test  tools  is  discussed.  Some  results 
' of  completed  analyses  are  presented;  however,  the 

interpretation  of  the  results  is  subjective  based  on  the 
author’s  experience  and  opinions.  The  material  in  this 
4.  ia  part  of  a book  chapter.  The  chapter  references 

«u:e  included  in  toto.  Peiragraph  numbering  follows 
*'  the  chapter  sequence. 
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A S'JBJZCrr/E  EVALUATION  OF  SELECTED  rRCGRAiM  DEVELOPMENT  TOOLS 


A tool  is  assigned  to  be  used  for  a specific  purpose  and  it  must  be 
handled  according  to  instructions.  Improperly  used  it  can  cause  more  damage 
than  it  is  worth.  Carpenters  learn  this  early  whan  they  are  taught  how  to 
use  a tool  such  as  aui  adze  for  squaring  off  logs.  The  adze  is  built  something 
lijce  an  axe  with  a horizontal  cutting  edge.  The  carpenter  straddles  the  log 
and  wields  the  adze  in  an  arc,  following  through  between  his  legs. 

Properly  used,  the  adze  rapidly  produces  a neat  plane  surface  on  a large 
log  or  plaidc.  Improperly  used  the  adze  is  dangerous.  A misstroke  caui 
cut  too  deep  and  spoil  the  surface  beyond  recovery.  More  important, 
a loose  grip,  a glancing  blow,  or  bad  aim  can  turn  the  cutting  edge  into 
the  carpenter's  leg  with  bloody  consequences.  The  carpenter  has  to  be 
convinced  that  the  benefit  of  using  an  adze  exceeds  the  cost  of  buying  and 
caring  for  one  and,  in  addition,  exceeds  the  cost  of  a mistake.  Net  surprisingly, 
the  adze  is  found  in  relatively  few  carpentry  kits. 

Tools  used  in  progrannning  projects  are  similar  to  tools  in  a 
carpenter's  kit.  Each  has  an  initial  procurement  cost.  Each  has  a 
continuing  maintenance  emd  update  cost.  Each  requires  proper  training 
before  use.  The  project  must  be  adjusted  somewhat  to  fit  the  tool. 

And,  in  spite  of  the  precautions  taken  by  the  toolmaker  ax:d  the  user  to 
avoid  misuse,  the  tool  can  cause  irreparable  damage.  A tradeoff 
analysis  comparing  the  costa  and  potential  risks  to  the  potential 
benefits  is  a prerequisite  to  a commitment  to  a specific  tool. 

There  aire  three  dominant  reasons  why  managers  authorize  the  use  of  tools : 
o increase  productivity 

o improve  quality 

o improve  resource  and  schedule  control. 

Really,  etll  three  are  only  different  aspects  of  a single  reason  for 
justifying  the  expense  of  a tool;  namely,  that  the  value  exceeds  the  cost. 

Tools  generate  value  by  increasing  the  useful  output  obtained  from 
a given  unit  of  work.  For  a program  systems  costing  $30,000,000  containing 
one  million  lines  of  code  Che  cost  of  production  would  be  $30  per  thousand 
lines  of  cods  or  $30/lCL0C.'^  Programmer  productivity  for  this  project  may 


The  scale  factor  KLOC  has  been  recommended  for  measuring  many  aspects  of  program 
development  because  it  is  based  on  a measurable  characteristic  of  actual 
programs.  Although  "lines  of  coda"  is  defined  in  various  ways  (to  include  or 
exclude  cemmants,  to  include  or  exclude  instructions  resulting  from  macro 
expansion,  etc.),  the  number  of  lines  in  a specific  program  can  be  determined 
simply  by  counting.  The  cost  elements  of  the  project  that  produced  that 
program  can  be  stated  as  $/KL0C,  (s,rson-montha/KLOC , defects/KLOC,  etc. 
facilitating  comparisons  across  products.  A key  supporter  of  the  lOOC  unit 
of  measure.  Capers  Jones  of  IBM,  has  shown  it  to  be  the  most  useful  conmon 
denominator  of  the  many  measures  used  in  IBM.  [9] 
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be  2500  instructions  per  programmer-year  or  0.4  programmer  years/KLOC.  (At 
$30, 000/person-year,  1000  person-years  expended,  400  programmer-years,  600 
support-and  manager-year.)  A similar  project  using  a tool  that  improves  the 
productivity  of  programming  and  testing  25%  should  spend  proportionately 
less.  Allocating  the  25%  saving  to  programming  amd  the  teat  portion  of 
support  (200  tester-years)  amd  leaving  everything  else  the  same,  the  project 
would  use  850  person-years  coating  only  $25,500,000  plus  the  cost  of  the 
aid.  In  most  organizations,  an  aid  coating  as  much  as  $4,500,000  would  be 
attractive  if  it  could  ensure  the  indicated  personnel  savings. 

An  improvement  of  25%  in  programming  and  testing  is  a very  laurge  improvement. 
Most  aids  do  not  promise  that  much;  in  fact,  moat  aids  do  not  pr™|Tse  any 
improvement.  The  value  of  an  aid  is  almost  always  based  on  consensus  opinion 
rather  than  on  quantitative  factors.  Each  aid  is  promoted  by  its 
originator  on  the  basis  of  common  sense  and  demonstration.  Thus , a technique 
such  as  structured  programming  is  presented  as  "a  logical  way  to  write  programs". 

A program  is  presented  in  two  versions,  one  structured,  the  other  not.  If  it  is 
not  obvious  to  the  observer  that  the  structured  version  is  better,  the 
demonstrator  will  ask  a few  c[uestions  about  the  prograun's  function.  The 

will  find  it  easier  to  answer  the  questions  by  reading  the  structured 
In  time  the  observer  will  convince  himself  that  structured  program- 
ming has  merit,  arriving  at  this  conclusion  at  about  the  same  as  the 

bulk  of  his  colleagues.  Most  techaiques  gain  acceptance  in  a similaLr  way. 
Initially,  someone  originates  the  technique  to  solve  a practical  problem. 

It  works;  so  the  originator  or  an  associate  uses  it,  becomes  devoted  to  it, 
and  proceeds  to  promote  it  outside  his  immediate  area.  For  a \Aile,  there  is 
little  response  other  than  admiration  for  a resourceful  problem  solver. 

Gradually,  a few  more  people  independently  pick  up  the  technique  until  a 
ground  swell  of  enthusiasm  sweeps  the  technique  into  wide  use.  At  this  time, 
each  new  user  adopts  the  technique  because  "everyone  is  using  it;  it  must 
be  good".  And  so  it  is.  The  technique  would  have  been  ed3£uidoned  eeurly  if 
it  were  inadequate.  Interestingly,  it  is  usually  after  the  widespread  acceptance 
of  the  technique  that  people  start  to  measure  its  effectiveness.  The  data 
obtained  can  be  used  to  suggest  areas  for  improvement  but  are  too  late  to 
affect  mamagement  decisions  regarding  the  technique's  value. 

The  pros  and  cons  of  each  tool  an  individual  might  elect  to  use  aire 
reasonably  self-evident.  The  tools  used  by  individuals  on  medium  and 
large  projects  (including  many  already  discussed)  interact  in  more 
complex  ways  auid  may  be  harder  to  evaluate.  For  instamce,  an  on-line 
prograsDing  system  \idiich  allows  a programmer  to  type  in  a source  program, 
edit  it  interactively,  and  compile  and  run  it  without  leaving  the  office 
may  cut  many  days  off  the  schedule  for  producing  the  unit  of  prograimming. 

Yet,  with  a hundred  prograamers  on  a single  project,  the  benefits  of 
the  interactive  terminal  may  disappear.  For  one  thing,  it  may  not 
be  economically  feasible  to  give  everyone  a terminal.  This  may  causa 
queues  to  build  up  at  each  terminal  and  may  force  some  people  to  leave  the 


office  to  find  an  available  terminal.  This  takes  away  mxich  of  the 
convenience  and  innediacy  of  on-line-  operation.  A project  that  depends 
on  interactive  with  no  backup  technique  may  be  out  of  business  when  the 

support  computers  are  not  available. 

Another  factor  affecting  the  large  groups  may  be  the  inability  of  one 
computer  to  handle  the  whole  workload.  As  will  be  seen  later,  the  integration  of 
a large  program  system  using  classical  methods  chews  up  many  hours  of  computer  time. 
The  interactive  terminal  user  generates  the  heavy  integration  ^md  test  woricload 
by  submitting  completed  program  for  integration.  He  then  must  coo^ete  with 

the  integration  woricload  in  order  to  get  time  for  editing  and  debugging 
interactively  the  program  unit  be  is  currently  working  on.  When  the  worJcload 
fills  the  computer,  the  terminal  response  time  gets  worse  and  worse  until 
it  is  no  longer  efficient.  When  the  workload  overflo«ra  the  computer,  a new 
problma  develops:  there  is  no  longer  a convenient  way  to  refer  to  all  parts 
of  the  system.  A new  communication  and  control  procedure  is  needed  to 
bridge  the  gap  fi»m  one  computer  to  the  other.  This  is  a quite  difficult  and 
e^^ensive  stepr  yet,  without  it,  the  basic  interactive  terminal  loses  some  of  its- 
c^ability.  It*s  like  a two-man  saw.  It  works  fine  on  small  trees  where  one-man 
can  get  on  each  end  of  tiie  saw.  But  very  large  traes  can*  t be  spanned  by  the  saw 
so  it  may  be  no  use  at  all. 

On  large  projects,  the  set  of  tools  that  was  appropriate  for  small  jobs 
must  be  augmented  by  tools  designed  just  for  large  projects. 

Host  such  tools  standard  ways  of  doing  things  and  standard  ways 

of  things  ap  everyone  on  the  project  can  understand  what  is 

going  on  and  so  davlations  from  tfaa  plam  are  easy  to  spot. 

There  are  three  areas  in  which  qgecdLal  tools  can  be  inqpartant  os  a 
large  projects 

o analysis  and  design 

o development  and  test 

o status  control 

Various  large  project  tools  are  covered  in  later  chapters  in  sufficiant 
detail  to  show  what  they  are  used  for,  how  they  affect  the  project,  and 

what  tradeoffs  should  be  made  when  evaluating  them.  In  most  cases,  the 
examples  are  adequately  described  elsewhere;  therefore,  rather  than  ea^lain 

mmf-H  tool  in  detail,  raferences  to  the  literature  are  provided.  The  general  stata- 
of-the-art  of  software  engineering  in  1976  is  discussed  by  Boehm  [10] . 

2.3.1  Analysis  and  Design  Tools 

The  steps  in  a project  (Table  2.1}  proceed  from  a general  problem 
statsBtent  to  a more  specific  reguirsments  document  showing  exactly  what 
the  system  must  do.  In  order  to  help  relate  the  various  relevant  aspects  of  the 


492 


problem  to  one  another,  analysts  can  draw  on  techniques  for  thinking  about  complex 
problems.  One  class  of  techniques  is  structural  modeling  [11].  It  uses  various 
methods  of  showing  the  relationships  among  problem  elements  and  weighting  the 
relationships  so  as  to  elucidate  the  structure  inherent  in  the  problem.  Given  the 
struct\ire,  system  dynamics  modeling  [12]  can  be  useful  in  pointing  out  behavioral 
aspects  of  the  problem.  The  amalysts  who  develop  the  detailed  requirements  select 
what  they  believe  is  the  minimum  set  of  necess^u:y,  feasible  items.  System  architects 
respond  to  the  requirements  by  specifying  what  will  be  built  - often  a subset  of 
the  requirements  statement  - and  how  it  will  look  to  the  user.  The  purpose  of  this 
step  is  to  specify  the  external  interfaces  of  the  systen  and  to  aillocate  system 
functions  to  sit.ier  side  of  the  interfaces.  Architecture  is  key  in  systems  which 
contain  or  communicate  with  existing  subsystems  which  constrain  the  design  of  the 
new  systKB  in  order  to  preserve  compatibility.  From  this  point,  the  internal 
design  of  how  the  new  system  will  behave  cam  proceed  step-by-step.  When  the 
design  is  detailed  enough  to  identify  program  units , the  units  are  assigned  to 
individiials  who  build  them  and  submit  the  results  for  integration,  test,  and  release 
to  operations,  followed,  in  most  cases,  by  a period  of  maintenance  and  improvement. 
The  large  number  of  people  involved  in  these  activities  makes  it  very  difficult 
to  have  direct  communication  between  all  of  the  implementers  and  the  key  analysts 
and  designers.  The  analysts/designers , knowing  that  they  aire  writing  specs  for 
people  they  may  never  meet,  need  a way  of  writing  requirements/specifications  that 
is  unambiguous  and  easy  to  understand.  The  method  should  be  particularly  good 
for  describing  interfaces.  It  should  be  easy  to  maintain  amd  modify  euid  should 
be  in  a form  that  supports  simulation.  The  components  of  such  a tool  ^u:e  an 
analysis/design  langxiage,  an  automated  specification  program,  a simulation 
capability,  and  a methodology  for  ensuring  complete,  consistent  results. 

Everyone  m\ist  learn  the  language  (vdiich  may  be  a programming  l;\nguage  or  simply 
a uniform  way  of  writing  specs)  . The  other  components  are  used  mainly  by 
the  analysts/designers.  The  automated  spec  is  a means  of  monitoring  programs 
written  by  the  implementers  to  see  that  they  conform  to  the  spec.  The  output 

of  this  program  can  go  back  to  the  implementers  to  help  them  see  where  they 

went  wrong.  The  simulator  exercises  a model  of  the  system  to  check  that,  if  all 
the  specs  eure  followed,  the  system  will  work.  It  is  the  designers'  proof  to 
the  implementers  that  the  spec  is  valid  and  should  be  obeyed.  NOw,  even 
though  no  single  designer  or  implementer  understands  the  whole  system,  there  is 
a single  baseline  spec  that  everyone  can  refer  to  in  order  to  resolve  questions . 

2. 3. 1.1  Analysis  and  Design  Languages 

Languages  for  system  analysis  must  work  in  the  managerial  as  well  am  the 
technical  areas  of  system  development.  They  must  cope  with  poorly  defined  user 
requests,  unsolved  technical  problems,  and  arbitrary  resource  constraints.  The 
indefinite  nature  of  these  aspects  of  analysis  tends  to  result  in  more  text 

segments  - explanations  of  assumptions  or  justification  of  decisions  - they  are 

found  in  finished  programs.  Still,  to  ia^rove  clarity,  traceability,  testability, 
and  to  maintain  positive  control  of  the  contents  of  the  requirements  at  all  times , 
system  analysis  languages  have  been  developed.  One  example,  based  on  the 
Problem  Statement  Language/Problem  Statement  Analyzer  (PSL/PSA)  originated 
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by  the  university  of  Michigan  ISPOS  project  [131  , is  contained  in  REVS  [14]  . 

REVS  is  a requirements  engineering  and  v^idation  ^stem  developed 
by  for  use  in  large  real-time  syst^.  The  language  component  of  REVS 
is  RSXi  deals  wiidx  system  entities,  messages,  activities,  and  networks 

(R-nets)  formed  from  the  possihle  paths  messages  may  follow  from  an  input 
interface  to  an  output  interface.  The  R-nets  are  constructed  according  to 
a small  number  of  rules  for  where  to  start  and  how  to  proceed.  The  result  is  a 
reqxiirements  statement  document  which  can  be  analyzed  for  cos^leteness  by  the 
REVS  si^port  facilities.  The  specific  TBST  system  luis  added  capability  for 
bulldinq  and  executing  simulations  faom  the  requirements  and  evaluating  the 
feesibility  of  meeting  the  system  performance  requiremmits . All  of  this  is 
esdaedded  in  a REVS  library  facility  ^diich  has  a flexible  inquiry  capability 
permitting  managers  to  investigate  project  status  from  a variety  of  viewpoints. 

It  is  significant  that  REVS  inclixies  "engineering"  in  its  title.  Eb^rience 
with  the  hardware  portions  of  large  systems  led  engineers  to  adopt  formal 
procedures  for  defining  the  project.  REVS  carries  engineering  discipline  over 
to  software  activities. 

A second  analysis  ^proach  is  SADT,  the  structured  analysis  and  design 
techntqiie  of  SofTech,  Inc.  Devised  by  Ross  [IS] , SADT  offers  a graphic 
Language,  as  it  were,  to  show  the  functions  to  be  performed  by  a systn. 

Great  eaiphasis  is  placed  on  the  identification  scheme  used  to  label  activity 
or  data  boxas  and  the  ^u3:aws  which  connect  the  boxee,  showing  inputs,  outputs, 
controls,  eud  proposed  mechanisms  for  ia^lenentatlon.  As  a result,  SADT 
permits  imiTtiple  hierarchical  views  of  the  system  to  be  correlated  with  one 
another.  Bach  viewpoint  supports  one  important  ampect  of  requirmnents  analysis. 

It  is  often  easier  to  develop  the  requiraments  in  one  class  if  the  requirements 
of  another  class  are  separated  cut.  Thus,  software  requiraments  to  satisfy  all 
msBibers  of  a user's  group  may  provide  one  viewpoint  and  a huge  list  of  functions 
to  be  parfoamed.  Different  reqioiraments  as  presented  by  a developer  reflect  concern 
for  oompecibility,  economy,  uaeebility,  and  other  factors  which  make  up  a much 
smal  ]er  list  of  functions.  SADT  permits  both  analyses  to  proceed  and  then 
provides  the  means  for  bringing  them  together  in  order  to  decide  on  the  list 
of  functions  to  be  adopted  which  may  be  a compromiae  between  the  two  viewpoints. 

The  reference  systam  that  supports  multiple  views  also  provides  the  data  needed 
to  check  the  raqalresients  document  for  completeness  and  consistency.  SADT 
differs  from  REVS  in.  that  the  analysis  and  design  stages  of  a project  are  both 
supported  by  SADT.  In  fact,  SADT  methodology  encourages  iteration  between 
requirmsmts  analysis  and  dosiga  at  ^ppropriata  points.  One  such  point  is  when 
analysis  stops  at  an  obstacle  which  could  be  rasioved  by  a design  decision. 

For  example,  detailed  requirements  for  the  content  and  visual  quality  of  a 
display  may  depend  on  knowing-  what  physical  device  is  to  be  provided  by  the 
ioplementer. 

Generally,  ell  analysis  and  design  techxiigues  including  SADT  postpone  such 
decisions  as  long  as  possible.  SADT,  however,  is  van'  flexible  in  this  respect 
whereas  some  design  techniques  assimw  that  analysis  is  complcta  before  design 
starts.  In  the  latter  group  are  the  design  methodologies  strongly  influenced 
by  the  structure  of  commercial  information  systams  departments.  Analysis  is  often 
done  by  a different  pert  of  the  organization  from  design  in  commerical  firms. 

The  analyst  skills  stress  subject  area  experience  (financial,  personnel]  and  design 
skills  stress  data  processing  experience.  The  departmental  split  permits  economic 
justification  for  new  work  to  be  done  by  the  analysts  to  avoid  make-work  tasks 
originating  in  the  programming  department.  Within  the  prograimaing  department 
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there  is  a need  for  tools  which  raise  the  performance  of  all  the  staff  up  to 
an  acceptable  average.  For  this  reason,  design  approaches  such  as  Jackson's 
[16],  Langefors' [17] , and  Warnier's  [18]  give  rules  for  converting  from  the 
design  language  to  actual  programs.  More  will  be  said  about  these  approaches 
in  Chapter  6. 

Languages  exclusively  for  program  design  have  not  achieved  wide 
acceptance.  Unlike  procedural  languages  such  as  PL/I  or  COBOL,  design 
liuiguages  do  not  have  an  agreed-upon  core  vocabul£u:y.  As  a result,  there 
is  no  one  design  l^ulguage  that  everyone  recognizes  as  the  best.  APL 
comes  close  because  many  people  know  it.  Nevertheless,  APL  is  used  more 
to  evaluate  designs  euid  to  create  oiodels  for  auialysis  than  it  is  to  document 
designs.  The  power  of  set  operations  available  in  APL  is  the  basis  of  SETL, 
advanced  by  Schwartz  [19]  as  a prograun  development  lauiguage.  Less  comprehensive 
proposals  such  as  the  Module  Interconnection  Language  (MIL)  suggested  by  OeSemer 
auid  Kron  [20]  tackle  less  of  the  development  process , concentrating  in  this  case 
on  formalizing  interface  designs.  It  may  be  that  no  standard  design  language  will 
sver  exist  because  it  is  not  needed.  The  evidence  for  this  comes  from  the  increase 
in  informal  program  design  languages  vAiich  mix  natvural  language  (e.g.,  English) 
descriptions  with  statements  in  a high-order  structured  programming  language  (HOL) 
such  as  PL/I,  ALGOL,  structured  COBOL,  or  structured  FORTSAN.  The  informal 
approach  builds  on  the  work  of  Oijkstra  [21,  22] , Wirth  [23] , Mills  [24]  and  others 
whose  emphasis  on  top-down  development  ituUces  good  use  of  mixed  format.  The  most 
successful  informal  languages  or  pseudo-codes  establish  a small  number  of  key  words/ 
commands  which  are,  in  effect,  the  names  of  the  basic  control  structures  of 
structTired  programming  [25] . Design  is  more  concerned  with  the  flow  of  control 
and  data  among  program  units  than  with  the  internal  behavior  of  the  units;  therefore, 
it  is  natural  and  effective  to  use  a design  language  that  highlights  control 
structures.  Flow  can  be  verified  by  deleting  the  natural  langxiage  and  con^iling  the 
remaining  statements.  It  is  a straightforward  step  to  proceed  from  the  pseudo-code 
design  to  a formally  coded  program. 

2 . 3 . 1 . 2 Automatic  Specification 

This  choice  of  an  informal  design  language  makes  it  more  difficult  to 
build  an  automated  specification  tool.  In  order  to  verify  that  a program 
agrees  with  its  spec , the  tool  must  be  able  to  recognize  corresponding  names , 
dimensions,  displacements,  and  the  like.  Informal  design  languages  do  not 
have  built-in  enforcement  procedures  to  require  all  progsainners  to  use  the 
same  conventions.  Obviously,  rigoroxis  rules  take  away  much  of  the 
informality.  Sooner  or  later,  however,  all  references  to  che  same  object 

in  a system  must  be  reduced  to  a common  base.  It  is  a function  of  design 

control  to  see  that  this  happens.  A compromise  approach  that  seems 
feasible  is  to  rely  on  a self-documenting,  compilable  design  as  the  input 
to  programmers  and  to  automated  spec  tools.  Informality  is  preserved  up  to 
a point.  The  designer  can  use  any  technique  for  draft  specifications. 

The  final  specification  must  be  free  of  natural  language  and  contain  only 
legitimate  statements  in  a programming  language.  The  spec  is  seif- 
documenting  if  appropriate  comment  statements  have  been  included  (in  the  working 
code  or,  better,  in  a prologue) . It  is  compilable  because  it  uses  HOL  statements 

exclusively.  At  this  stage,  all  system  references  must  agree  with  the  standard 

references  in  the  glossary  of  system  names  emd  synonyms.  As  a design,  however,  the 
spec  lacks  the  detail  necessary  to  make  the  program  units  function. 


Kow  an  automacad  spec  tool  can  con^are  a completed  program  unit  to  the 
spec  and  determine  that,  for  each  control  statement,  identifier,  or  variable 
in  the  spec,  there  is  a corresponding,  correctly  named  entry  in  the  program. 

It  can  also  flag  program  references  that  have  no  counterpart  in  the  spec. 

Deviations  such  as  this  are  brought  to  the  attention  of  both  the  programmer 
and  the  designer  (or  design  control  group)  to  ensure  that  the  deviation  is 
explained  or  corrected  [26] . 

As  more  is  learned  about  proving  the  correctness  of  programs,  it  is 
likely  that  both,  designs  and.  coded  program  units  will  be  checked  for 
correctness.  TO  do  this,  assertions  regarding  the  intended  behavior  of  a 
program  most  be  included  is  the  ^ec.  A proof  mechanism  built  into 
the  autoaiated  spec  tool  would  test  the  assertion  by  carrying  out  the  logical 
steps  described  by  the  spec  or  the  con^leted  program.  The  results  would  be 
particnlarly  useful  in  large  programs  with  many  internal  branch  points.  It 
is  easy  for  the  designer  to  get  confused  in  such  circumstances,  onf ortunatsly , 
the  state-of-the-art  of  correctness  proofs  in  the  1970s  does  not  span  large 
enough  program  systsms  to  solve  the  designer's  problem  [27] . future 
promise  of  proof  tools  nevertheless  justifies  the  adoption  today  of  tbm 
'^  •***••*  o£  mmifintj  appropriate  assertions  as  an  aid  to  program  verification. 

Once  made,  the  assertions  will  bs  a useful  aid  to  inspectors  who  examine  the 
design  and  code  at  appropriate  points  in  the  project.  The  assertions  will  clarify 
the  prograsmer's  intent  better,  perhaps,  than  the  program  text. 

2.2.1.a  SIPO 

A procedure  for  preparing  design  specs  so  thsy  are  readable,  well- 

organized,  systasmtic,  to  programmers  in  HIPO  [28] . HXPO  is  a 

tachnique  for  displaying  Hierarchy,  Uput,  Processing , and  Output. 

It  parallels  top-down  design  by  proceeding  from  overview  to  detail  (Fig.  2.8)  . 

TSa  types  of  charts  are  used.  One  is  a tree  s'teucture  representing  the  program 
systaa  structure  (Fig.  2.9e)  . It  is  the  index  to  the  process  diagrams  (Fig.  2.9fa) 
which  show  inputs,  pcocass,  and  outputs  and  below  if  necessary  additional  infor- 
mation (such  as  assertions,  manageowst  guidance  as  to  schedules,  resources,  etc., 
or  guidance  such  as  standards  or  macros  to  be  used) . The  detail  diagrams 

can  be  filled  in  with  Qiglish  or  pseudo-code  or  POC  statements.  The  purpose  of 
HXPO  is  to  provide  a graphic  view  of  a complex  system  without  getting  lost  in  the 
detail.  EXPO  does  not  give  guidance  in  bow  to  design.  It  singly  documents  the 
design.  la  this  respect  HIPC  is  less  sophisticated  than  the  design 
tacbniques  aLlready  mentioned.  It  is  also  simplar  and  cheaper.  Being  easy 
to  use  ia  any  enviranmettC,  htpo  can  help  overcome  the  cuuiiHuni cation  problesm 
that  oftan  hurt  projects. 

HIPO  is  quite  similar  to  an  approach  taken  by  the  System  Development 
Corporation  tdian  they  prepared  a planning  guide  for  the  use  of  the  O.  S. 

Naval  Cruimand  Systssw  Si^port  Activity  [29].  Prepared  in  1965,  the  guide 
gave  Navy  project  leaders  instructions  for  managing  a complete  prograsmiing  life 
cycle.  In  most  respects,  it  is  valid  today.  It  is  mentioned  here  because  the 
main  technique  for  showing  the  steps  In  the  life  cycle  is  ramerkably  like  HIPO. 

The  life  cycle  is  treated  as  a sequence  so  no  tree  structure  diagrams  appear; 
however,  each  datail  diagram  looks  Ilka  a HIPO  chart  with  the  addition  of  cost 
and  anvironawnt  data  (Fig.  2.10) . It  ia  elaar  that  the  HIPO  technique  ia 
useful  and  flexible  and.  Indeed,  meets  a need  fwr  more  systsmetic  design 
documentation  [30] . 
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2. 3. 1.4  Modelling  & Simulation 

A design  language  ^uld  some  means  of  design  verification  give  a degree 
of  control  over  the  functions  of  the  system  being  built.  These  tools  do  not 
deal  with  system  performance.  Modelling  and  simulation  eure  required  to 
predict  performance.  Modelling  consists  of  building  an  analytical  or  a 
procedural  model  of  the  system  design.  Simulation  consists  of  feeding 
realistic  inputs  to  the  model  and  observing  the  model's  behavior. 

In  the  early  stages  of  design,  analytical  models  can  describe  the  intended 
behavior  of  various  portions  of  the  system.  An  euialytical  model  is  a set  of 
mathematical  equations  euid  logical  expressions  which  represents  system  structure 
and  data  flow.  The  algorithms  used  to  develop  the  equations  are  often  patterned 
after  existing  systems  that,  hopefully,  resemble  the  planned  system.  Thus,  the 
paging  behavior  of  a current  operating  syston  may  be  adopted  to  describe  how 
memory/disk  traffic  will  look. 

Analytical  models  represent  the  design  in  equations  \diich,  if  correct  and 
solvable,  produce  repeatadsle  results  rapidly.  In  general,  the  functions  of  a 
single  program  unit  can  be  modelled  emalytically  but  a program  system  cannot. 

Thera  are  too  many  branches  in  the  system  resulting  in  random  (or  stochastic) 
behavior  that  can  only  be  guessed  at.  Also,  the  programs  consist  of  discrete 
events  which  the  analytical  modal  approximates  by  continuous  equations.  As  a 
result,  analytical  models  alone  are  unsatisfactory  predictors.  They  are  best  used 
as  components  of  a procedural  modal  or  as  quick  tests  to  obtain  ballpark  estimates 
of  performance.  The  latter  are  very  important  since  it  is  sufficient  to  know  that 
a design  is  in  the  right  ballpark  \dien  you  must  decide  to  accept  it  or  reject  it. 

The  greatest  strength  of  analytical  models  is  in  predicting  the  effect  of  a 
design  change  on  an  existing,  well-documented,  communications-oriented  system. 

In  this  situation,  response  time  is  a key  criterion.  Good  data  about  input  arrival 
times  euid  subsystem  service  times  often  exists  in  running  systems  so  that  reasonable 
assessment  of  a design  change  is  possible.  The  same  methods  are  leas  useful  in  a 
new  system  because  of  the  uxiavailability  of  good  arrival  and  service  time 
distributions.  One  usually  assumes  some  standard  arrival  ^md  service  time 
distributions  and  settles  for  ballpark  answers.  The  margin  of  error  is  high  but 
the  information  obtained  will  still  help  guide  the  designer  to  better  solutions. 

An  accepted  design  must  be  controlled  to  tighter  tolerances.  Here  is 
where  the  procedural  model  comes  in.  It  is  constructed  exactly  like  the 
system  design.  For  each  element  in  the  design  there  is  a corresponding 
element  in  the  model  and  for  each  control  sequence  in  the  design  there  is  a 
corresponding  control  sequence  in  the  model.  In  fact,  if  the  design  is 
written  in  pseudo-language  or  POL,  it  is  already  the  skeleton  of  the  model. 

To  complete  the  model,  all  you  do  is  enter  performance  factors  for  each 
element  or  control  sequence.  Early  in  the  design  stage,  it  is  necessary  to 
guess  at  the  performance  factors.  Educated  guesses  as  to  the  path  length, 
real  memory  size,  and  virtual  working  set  size  of  each  section  of  the  program  are 
best  mads  by  using  similar  existing  programs  for  reference.  As  the  project 
advances,  the  early  estimates  can  be  Improved  by  the  progranmers  amd, 
eventually,  exact  measurements  can  be  made  of  completed  program  units.  The 
model  gets  better  as  it  matures.  During  early  design,  results  provide  only 
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' ballpark  figures,  perhaps  SO  - 100%  from  the  true  values.  By  the  tlma  the 

I system  is  ready  for  delivery,  simulation  can  predict  consistently  within  5->10%  of  ^ 

I actual  performance.  In  complex  systems,  simulation  results  may  sometimas  surprise 

the  designers  because  they  exhibit  behavior  due  to  subtle  systan  interactions  the 

designers  never  thought  about.  Guest,  I<ai,  and  Loyear,  in  their  report  on  the 

' extensive  simulation  of  the  intelligent  controller  subsystem  in  a banking 

system  [311  , show  that  such  surprises  aire  usually  helpful. 

The  drawback  of  modelling  and  simulation  is  its  high  cost.  Even  when  the 

design,  serves  as  the  model  there  is  a large  coat  associated  with  obtaining 
perfomancs  factorsr  updating  tiis  models  and.  running  some  number  of  simulations. 

There  are  only  • fete  people  syatesi  design  so  that  the  added  cost  of  system 

analysis  the  design  stage  appears  to  be  teubling  project  cost.  In  fact, 

like  most  of  the  best  tools,  simulation  requiras  an  early  es^enditure  to  avoid 
a mrt,Th  larger  expenditure  later.  Thus,  $20  • 50000  worth  of  systems  analysis 
Mh'tt-h  selects  a design  altmnatlve  tiiat  does  the  customer's  job  can  avoid 
$1,000,000  worth  of  rework  on  a system  that  fails  to  pass  its  acceptance  test. 

6 gives  examples  of  the  types  of  questions  simulation  can  answer 

justi^  its  cost  in  complex  systems. 

2. 3.1.5  Bencfamaxkm  and  Srototypes 

Msnagers  who  are  frighten**  by  high  iititial  costs  sometimes  try  to  get 
''  perfbxmance  date  by  other  means.  The  most  popular  metiiad  is  the  benchmark  [321  • 

TeiH « i m not  a mr^  desigu  tool.  NSither  is  it  inexpensive.  A benchmark 

consists  of  e «^*^n**  set  of  jobe  with  input  and  database  givmi  and  output  alther 
known,  or  easily  A benchmark  test  is  e timing  run  in  which 

standard  joh  is  executed  against  the  clock.  Obviously,  to  be 

valid  the  test  most  *«*«~n**i  actuel.  programs.  But  there  are  no  programs 
available  *«^"g  the  design  stager  therefore,  « bmxchmark  at  this  stage 
imnmf  bw  of  programs  ficae  some  systam.  The  only  condition 

rmAmr-  which  ths  benchmark  will,  predict  the  perfisrmanee  of  the  nee  system 
is  that  the  nee  system  be  tdentlr»T  to  the  old  one.  Tat,  if  you. 

tiiat  to  be  the  case,  you  do  not  have  to  run  the  benchmark  at  all. 

A ^ ^ version  of  tiie  planned  system  [331  • It.  may  be  a subsat 

o£  planned  system  merely  functions.  In  this  form,  it  is  often 

usaii  to  check  a particular  design  decision  at  the  module  level.  The  prototype 
B^y  also  be  con^letely  from  the  planned  system  “ a "quick  and  dirty* 

version  - which  is  going  to  be  discarded  after  it  is  studied.  Where  the  new 
systam  is  different  from  the  old  one,  two  arguments  arc  advanced  for  using 
benchmarks.  Both  are  fanltys 

o it  is  to  extrapolate  from  old  data.  The  only  case  this 

applies  is  when  the  new  system  has  the  sane  structure  in  detail  as 
the  one  but  has  program  units  of  diffarant  size.  Itaw  systems  of 
type  are  unrealistic  investmunts.  There  probably  are  none. 

o it  is  possible  to  extrapolate  from  a prototype  of  the  new  system; 
namely,  actual  code  of  a llmitad  portion  of  the  new  system.  This 
approach  works  to  a limitad  degrae  when  the  prototype  represents  the 
bulk  of  ths  execution  (say,  80-90%  of  the  path  langth)  of  the  new  | 

system.  Few  complex  systems  possess  such  a ti^t  kernel. 

f 1 

The  accuracy  of  simulators  aarly  in  tha  design  stage  is  not  actually  known 
because  there  is  never  a fully  axacutable  version  of  the  system  represented 
at  that  tiaa  in  the  model. 
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2. 3. 1.6  Ex^apolation  o£  Softwar*  Data 

The  real  reason  these  argrs&ents  fail  is  that  extrapolation  in  the 
software  business  is  very  unreliable,  when  you  use  program  A as  a reference 
to  predict  something  about  program  B,  you  must  rely  on  A and  B being  largely 
alike.  But,  as  noted  eibove,  no  one  wants  to  spend  a lot  of  money  to  get 

a B that  is  just  like  the  A he  already  has.  B must  have  added  value  due 

to  more  functions  or  better  cost/perf ormeuice . Usually,  B is  either  larger 
than  A or  different  from  A.  Fig.  2.11  illustrates  that,  while  you  could  predict 

B from  A if  they  were  alike,  you  cannot  predict  B from  A when  they  differ,  and 

you  cannot  predict  B accurately  idien  it  is  larger  than  A.  The  last  case 
is  the  most  cosmon  and  the  most  likely  to  cause  e]q>ensive  errors.  The 
picture  shows  B twice  as  large  as  A but,  following  the  square  law  of 
complex  systems,  B is  four  times  as  complicated  as  A.  In  a sense,  when 
you  use  A as  a reference,  you  are  ignoring  75%  of  the  problem.  . This 
picture  appliefs  whether  you  are  predicting  performance,  estimating  workload, 
or  setting  schedules.  Overreliance  on  an  existing  reference  system  will 
lead  to  large  understatements  of  results. 

A couple  of  examples  show  what  has  happened  in  the  past: 

o A is  similar  to  B - one  case  where  it  is  worthwhile  to 

copy  A is  in  the  compiler  business.  An  experienced  crew 
using  the  same  compiler  design  to  produce  compilers  for 
different  machines  can  generally  achieve  performance 
proportional  to  the  speed  and  storage  of  the  target  machines. 

o A is  different  from  B - at  one  stage  of  development,  the 
real  time  operating  system  (RIOS)  for  Apollo  was  about  the 
same  sire  as  OS/ 3 60.  RTOS  used  a speciatl  algorithm  for 
routing  urgent  traffic  whereas  OS/ 360  made  its  fxill  function 
available  to  eU.1  tremsactions . RTOS  appeared  to  execute 
faster  than  OS/360  but,  in  fact,  the  two  systems  were  not 
directly  comparable.  Each  had  its  own  user  environment. 

o A looks  like  B but  is  acttuilly  smaller  - a project  to  convert 
an  existing  program  from  a small  machine  to  a larger,  newer 
machine  expected  to  get  a performance  improvement 
of  4:1  based  on  the  ratio  of  machine  speeds.  Additional 
functions  were  added  and  the  new  program  was  written  by 
inexperienced  coders.  It  ended  up  5>6  times  larger  than  the 
old  program.  The  actual  performance  on  the  larger  machine  was 
1/10  that  of  the  small  machine. 

Benchmarks  are  poor  design  tools  because  they  are  poor  representations 
of  the  system  to  be  built.  When  used  as  aids  to  computer  selection,  this  major 
disadvantage  gets  cancelled  out.  When  several  bidders  are  given  the  same 
problem,  they  all  start  from  scratch.  The  best  test  performance,  while  a 
poor  indicator  of  ultimata  system  performance,  may  be  an  acceptedsle  indicator 
of  the  best  proposal.  In  this  case,  the  benchmark  is  not  expected  to  be  a 
precise  model  of  the  new  system.  As  long  as  it  is  in  the  same  class  (A  looks 
like  B) , it  can  be  used  to  distinguish  among  bidders.  Thus,  it  is  possible 
for  government  agencies  and  computer  companies  to  build  a library  of  standard 
benchmarks  to  be  used  for  comparing  one  generation  of  computers  to  a 
predecessor  or  to  compare  one  line  of  equipment  to  a competitive  line  [34] . 
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A«  a design  and  development  tool,  modelling  and  simulation  are 
substantially  more  useful  than  any  level  of  benchmarking  or  prototyping  as 
long  as  the  model  is  faithful  to  the  design  and  is  kept  current.  This  is 
feasible  when  the  model  is  the  byproduct  of  design.  The  model  can  be  kept 
current  as  a byproduct  of  design  control.  In  these  circrmstances , one  use  i. 

of  the  model  (say,  20  runs  to  evaluate  alternate  design  proposals)  can  be 
expected  to  take  10-15  analyst'*weeks  and  30-60  hours  of  C?U  use  (not  counting 
connect  time  when  interactive  terminals  are  used  during  simulation) . One  i . 

benchmark  takes  about  the  same  number  of  analyst  weeks  and  requires  considerably 
more  compucer  time  - on  the  order  of  100-200  hours  of  the  main  C?U  and  almost  ' 

as  much  for  a driver  (to  supply  an  input  stream  from  a second  CPU) . The  { 

development  cost  of  a model  is  spread  across  its  life  which  can  extend  for  the 
life  of  the  product  it  represents.  In  fact,,  the  most  profitable  use  of 
mtxtels  is  oftea  tm  tone  or-  upgrade  installed  systasw.  The  development  cost 
of  a bsnchmask  is  a one-tijae  charge  incurred  for  each  benchmark  cast  unless  a 
library  standard  is  uasd. 

2.3.2  Ssvelopmsnt  and  Tsst  Tools 

Two  major  typss  of  dsvelopment  and  teat  tools  are  important: 

o tools  that  ijter«»e.<ie  individual  prcductivity  and  quality 

a too3.s  that  iacraase  project  productivity  and  quality 

When:  the  two  types  of  tools  are  coord inatsd,  the  total  value  of  the  resultinq 
development  support  system  is  enhanced. 

Naixy  paepi*  have  to  the  state~of-tfa*-est  is.  programming 

cachnology.  Ace^tance  by  bnsinsasman  of  a caaprahansive  set  of  consistent 
tsehaiqisne  fiOXlcMed.  pssctlcal  damnnstratlons  such,  as  those  caportad  by 

letiig  and  Tarry  Baker  of  the  IBK  Tedsssl  Systams  Division..  [351  ^ Sources 
variety  of  books  [22^24^  36-48]  and  other  materials  [49-57]  , discnasing 
improved  technologies  (XBD  at  a baatc  level,  m 1976,  TST  included: 


o stxuetsrad  pcograaaKlsg 

a code  reading 

o top-down  development  and  step-wise  refinement 

o structured  design 

o pseudo-code  program  design  language 

o BXPO 


o chief  prograoBsr  tsaas 
a dsvelapaest  support  Libraries 
o strueturad  walkthroughs 

o inspections 


Tbs  csfsrancaa  contain  a ssapls  of  IBM  itams  readily  available  to  this  author. 
Tbsra  ara;  of  course,  other  sources  for  vendor  manuals  and  reports. 
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Each  yaar,  soma  of  these  items  will  become  so  common  in  daily  use 
that  they  will  no  longer  be  considered  "improved"  technologies. 

Mew  itws  will  constantly  be  identified,  proven  by  experiment,  and  added  to 
the  IPT  list  to  maintain  the  stream  of  progress. 

Uhen  these  tools  are  all  used  on  a project,  the  net  result  is  expected 
to  be  an  improvement  in  both  the  individual  and  the  project  aspects  of 
productivity  and  quality.  The  reason  is  that  the  methods  tend  to  build 
quality  in  from  the  beginning,  eliminating  the  need  for  much  of  the  costly 
scrap  and  rework  that  characterizes  the  tail  end  of  many  projects.  Such 
results  are  "expected"  but,  unfortunately,  not  proven.  It  is  improbable 
that  a sound,  repeatable  experiment  will  ever  be. conducted  to  test  the 
hypotheoeis  that  is  better  than  old  methods.^  nevertheless,  experienced 
managers  are  adopting  IPT  widely  because,  at  a minijoum,  they  find  it  decreases 
their  risks.  Without  going  into  ouch  detail,  it  is  possible  to  see  why. 

2. 3. 2.1  Structured  Programming  (SP)  vs.  Unstructured  Programming 

Managers  are  not  comfortable  managing  products  they  cannot  understand. 

They  can  understand  structured  code.  They  know  that  most  discussion  about 
programs  takes  place  at  the  level  of  functions  which  are  often  recognizable 
as  substructures  of  a good  program.  Therefore,  SP  makes  it  easier  for  a 
manager  to  talk  to  a programmer.  SP  makes  it  easier  for  a writer  to  document 
the  prograuB.  SP  makes  it  easier  for  a diagnostician  to  trace  an  error.  SP  makes 
it  easier  for  a programmer  to  write  the  problem  correctly  in  the  first  place. 

2. 3. 2. 2 Code  Reading  vs  MO  Code  Reading 

It  is  well  know  that  the  originator  of  a document  is  a poor  proofreader 
of  the  result.  Par  one  thing,  the  writer  retains  enough  of  the  material 
in  his  muBory  that  that  he  sees  what  he  expects  to  see  rather  what  is 

written.  [Cius,  the  redundant  "that"  in  the  previous  sentence  is  easily 
overlooked  in  proofing.]  Retention  is  even  more  pronounced  when  copying  data, 
particularly  numbers.  A keypuncher  is  so  likely  to  make  the  same  error  on 
the  second  reading  of  a source  document  that  it  is  normal  to  use  a second 
person  to  verify  the  cards  produced  by  the  first.  The  same  is  true  in  publishing 
where  an  editor  proofs  the  galleys  prepared  by  a typist. 

Proofreading  by  the  buddy-system  was  also  common  in  the  early  days  of 
computing  when  programmers  wanted  help  or  wanted  to  show  off  good  code. 

The  practice  ehbed  as  programming  workload  grew.  Programmers  no  longer 
felt  they  could  afford  the  time  to  help  others.  Code  reading  came  back 
into  style  when  it  was  pointed  out  that  the  practice  (a)  catches  more  errors 
than  many  other  methods  and  (b)  leads  to  better  structure  and  more  understandable 
programs. 


A controlled  experiment  would  require  at  Laast  two  independent  developments 
of  the  same  large  scale  system.  Such  an  experiment  is  uneconomic.  Further- 
more, the  normal  variance  in  programmer  skill  can  cause  major  productivity/ 
quality  differences  regardless  of  tools  used  by  two  independent  groups.  If 
one  group  does  the  job  twice,  the  experiment  also  fails  because  the  knowledge 
gained  on  the  first  version  will  dominate  the  factors  contributing  to 
productivity/quality  in  the  second  version. 
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The  value  of  independent:  verification  has  been  extended  to  the  analysis  and 
design  activity  in  SADT.  An  integral  part  of  Boss'  method  is  the  thorough  review 
of  each  graphic  document  leading  to  an  exchange  of  corrections , modifications , 
and- suggestions  between  the  reader  and  the  author. 

2. 3. 2. 3 Top-Down  vs  Bottom-Up 

Implementation  from  the  top-down  verifies  each  aspect  of  the  design  as  it  is 
developed.  If  the  design  is  faulty,  the  faults  can  be  dealt  with  immediately 
without  propagating  effects  to  other  parts  of  the  system.  Purthermore,  the 
cost  of  getting  the  design  right  is  minimized  since  there  are  only  a small 
Qf  people  involved  ng  the  design  stage.  Errors  in  a bottom-up 
iBplasMn-tation  appear  when  code  is  complete  at  which  time  the  maximum  number 
of  people  are  on  board.  are  handled  in  either  approach  by  reiterating 

through  the  affected  parts  of  the  design.  The  results  of  making  a change  are 
gere  easily  controlled  in  top-down  design  since  all  affected  parts  exist  and  no 
guesswork  is  required. 

2. 3. 2. 4 Structured  Design  vs  unstructured  Design 

Structured  design  refers  to  the  conscious  control  of  binding  and  coupling. 

aTs  tTuly  independent  and  dependant  modules  have  ea^licit 
' interfaces  parameters.  As  a result,  programmers  can  work  on  individual 
without  fear  of  modifying  other  people's  work. 

Structured  ign  also  relies  on  explicit  development  of  models  of  system 
data  system  functions.  Convlataness  can  b*  checked  by  mapping  ths  data 
onto  th*  function  models.  Maintenanca  is  also  in^roved  by  virtu*  of 
th»  clean  interfaces  facilitats  diagnosis  and  tend  to  pinpoint  th*  sourca 

of  a problesu 

2. 3. 2.5  Program-Design  language  vs  Preer-Fom  Design 

One  project ‘that  got  into  trouble  was  audited  by  a teem  of  experts  to  sea 
what  was  required  to  cea^leta  the  job.  There  ware  soma  fifteen  people  in 
four  groups  worieing  on  what  they  said  was  a structured  top-down  design. 

Th*  auditors  found  each  group  using  a different  technique  — flowcharts,  PL/I, 
prose,  assembly— language  — for  design.  No  one  knew  vAiat  the  others  were 
doing.  Nor  could  th*  auditors  figure  out  the  design  status.  Here  was  a clear 

where  th*  of  a uniform  design  language  resulted  in  a project  failure. 

The  audit  team  recaanendation  was  adopted:  the  project  was  taken  away  from  the 
fifteen  people  and  reconstructed  by  another  group  at  another  location.  The 
procedural  discipline  of  a program  design  language  avoids  such  trouble. 

2. 3. 2. 6 HIPO  vs  Flowchart 

The  combination  of  functional  data  with  procedural  and  managerial 
data  gives  everyone  on  a project  a better  view  of  what  to  do,  when  to  do  it, 
who  is  affected,  and  where  each  activity  fits  into  the  overall  project. 

Simple  flowcharts  describe  program  structure  but  not  project  structure.  Siaqple 
activity  charts  (PEST  charts)  show  project  structure  but  not  program  structure. 
irrpQ  or  its  equivalents  show  both. 

2. 3. 2. 7 Chief  Prograamer  Teams  vs  other  Organizations 

Team  operations  allocate  tasJcs  on  the  basis  of  skills  and  focus  attention, 
via  the  chief  prograamer,  on  top-down  Implamentation.  It  is  easier  to  use  IPT 
tp  4 organization.  CoaoRinication  is  slaipler,  takes  less  time,  and  is 

aore  effective  in  a team.  The  team  is  a hierarchy  headed  by  the  chief 
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prograinmsr  but,  because  it  is  small,  personal  relations  can  be  excellent. 

Small  teams  cm  also  operate  effectively  without  a chief  programmer  because 
coonunications  lines  are  short.  Nevertheless,  there  is  a lack  of  focus  on 
the  hierarchical  nature  of  the  program  system  nkJcing  it  somewhat  more 
difficult  to  Quuiage  a top-down  implementation.  Large  groups  tend  to  lose 
the  focvis  altogether  amd  act  as  collections  of  peers  with  individual  goals 
instead  of  a team  with  a single  goal. 

2. 3. 2. 8 Development  Support  Libraries  vs  Private  Code 
When  work  in  process  is  collected  in  an  on-line  library  rather 

kept  in  the  programmer's  desk  several  benefits  appear.  The  official  record 
of  the  system  is  accessible  to  all  programmers  and  managers.  Less  is 

spent  handling  card  decks  and  tapes.  Code  reading  is  supported  by  current 
output  listings.  Interactive  code,  debug,  and  test  aids  can  be  installed. 

Skilled  libreurians  can  enter,  maintain,  and  retrieve  libreury  records.  They 
can  ^d.so  run  tests  as  requested  and  distribute  the  output.  Ultimately, 
delivery  of  the  progrzun  package  can  be  made  automatically  from  the  library. 

2. 3. 2.9  Structxired  walkthroughs  vs  Checkpoints 
A checkpoint  is  a date  in  a project  schedule  when  an  activity  is  to  be 

completed  and  managesient  intends  to  check  that  it  is  done.  To  avoid  the 
problems  caused  when  an  activity  is  complete  but  wrong,  the  structured 
walkthrough  converts  the  checkpoint  into  a careful  examination  of  the 
completeness,  accuracy,  and  general  quality  of  the  work  product.  The 

reviewers  are  often  part  of  the  development  team.  The  programmer  who  did  the  ^ 

work  under  review  walks  them  through  the  program  step-by-step,  explaining 
how  the  program  works  and  how  the  teat  plan  will  verify  the  program  functions. 

A sample  input  is  traced  through  the  program.  The  reviewers  then  critique 
the  work  and  recommend  what,  if  anything,  the  programmer  should  correct,  chamge, 
or  restudy.  Different  aspects  of  th*  program  draw  attention  at  different 
> stages  of  development  as  shown  in  Table  2.2.  The  whole  process  is  quite  informal. 

It  relies  on  peer  pressure  from  the  reviewers  on  the  developer  to  produce  a high 
quality  product. 

2.3.2.10  Inspections  vs  Structured  Walkthroughs 
The  success  of  a structured  walkthrough  depends  on  the  personal  relations 

within  a project.  Some  walkthroughs  are  more  successful  than  others.  The 
more  formal  technique  of  inspection  overcomes  the  vzuriability  of  walkthroughs  [46] . 
Inspections  use  a review  team  chaired  by  a moderator  who  orgauiizes  the  review 
and  reports  the  results.  The  team  members  aure  the  designer,  implementer,  and 
tester  of  the  program  being  inspected  plus  other  technical  experts  or  affected 
parties  as  necessary.  Usually,  four  people  are  plenty.  When  an  activity  satisfies 
the  project  exit  criteria  for  design  cosipletion  or  code  conqpletion,  the  moderator 
conducts  an  inspection.  The  inspection  procedure  follows  a set  pattern  in  which 
the  design  (at  inspection  1^^)  or  the  code  (at  inspection  I^)  is  read  amd  analyzed 

for  logic  and  accuracy.  Errors  aure  identified  (using  a checklist  to  help) . 

Based  on  the  error  rate,  the  moderator  can  either  authorize  the  activity  to 
proceed  or  send  it  back  for  rework  (Fig.  2.12).  Inspections  are  an  explicit 
management  tool.  They  capture  quality  data  for  management  amalysis  and  they 
resxilt  in  management  decisions  affecting  project  schedules.  Whereas  walkthroughs 
were  said  to  occur  at  checkpoints,  inspections  occur  at  milestones.  The  distinction 
between  a checkpoint  and  a milestone  is  that  the  former  verifies  that  an  event 
occurred  and  the  latter  verifies  the  occurrence  and  requires  a management  decision 
to  proceed  or  modify  the  project  plan. 
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2.3.2.11  Suppom  Tools 

Much  of  tha  basic  library  of  a program  davelopmant  organization 
providaa  siqiport  to  tha  progranmars.  Compilars  and  assaaiblars  support 
tha  dacision  to  usa  a hi^ar-laval  languaga.  Traca  programs,  maasurraant 
aids,  and  utilitias  stq^rt  program  analysis,  manipulation,  and  storaga. 

Ovar  and  abova  thasa  coomon  tools  are  program  sy stems  designed  solely  to 
maJca  progrananing  easier  emd  more  reliable.  The  general  characteristics  of 
these  sv^port  systems  are: 

o interactiva  facilities  both  to  help  the  programmer  con^se 
a program  and.  to  coach  the  programmer  in.  location  procedures 
and  usa  ot  the  tool 

o displays  supplementing  keyboard  terminals  to  provide  quiet 

access  to  a limited  segment  of  a program  or  document  plus  the 
ability  to  access  to  a hard  copy  of  th»  full  listing  when  needed. 

o central  files  of  project  control  information,  data  set  definitions, 
where  used,  tables,  etc. 

o library  suvlces  for  storing  programs,  isolating  woric  in  process 
tram  released  prograsm,  and  retrieving  programs  or  program 
segments  for  authorized  users. 

o text  preparation  services  for  editing  programs  and  preparing 
system  documentation. 

o test  esacutioa  facilities  based  on  a simple  mrnBand  systn 
for  selecting  test  cases  and  programs  from  the  library  and 
running  e sequence  of  tests. 

o fari Titles  for  simulation  and  performance  meesurwent 

o facilities  for  generating  teat  cases 

o facilities  to  collect  system  measurements  emd  user  statistics 
for  management  reports. 

Attsmpts  to  organize  all  thsss  functions  in  a single  system  lead  to  extremely 
large  support  facilities  which  become  development  probless  themselves. 

Therefore,  it  has  been  rscognizsd  that  the  best  siqpport  system  is  a coUaction 
of  individual  tools  which  hsvs  been  designed  to  work  together  within  a canon 
architacture.  It  should  be  possible  to  link  the  tools  together  auccmeticsUy 
to  produce  a useful  sequence  of  operations  such  as  program  entry,  cenpile, 
test  ease  genera tioa,  link  adit,  load  and  execute,  file  program  and  tests 
for  future  use,  prepare  and  release  results. 

By  planning  tha  approach  around  separate  tools,  the  full  capability 
of  e ST^port  systsm  can.  be  obtained  by  a development  shop  which  cannot 
afford  to  build  its  own.  Pomeroy  [581  listed  some  of  the  tools  that  were 
conereially  available  in  1972.  Sines  then  the  list  has  expended  to 
include  various  on-line  systvse,  display  processors,  structured  programming 
aids,  syntax  checkars,  data  baas  managers,  text  proesaaors,  teat  generators, 
CQsmand  systems,  and  reporting  aids  [59].  The  major  drawback  is  that  these  tools  j 
are  seldom  designed  to  a consistent  architecture  even  within  a single 
vendor's  catalog.  On  tha  other  hand,  it  is  not  hard  to  modify  the  output  of 
one  to  satisfy  the  input  raquiraments  of  another. 
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An  organization  with  many  programmers  simultaneously  coding,  debugging, 
testing,  auid  shipping  programs  will  have  one  or  more  computers  dedicated  to  the 
support  system.  Most  organizations  will  have  to  use  their  operational  computer. 

In  either  case,  the  preferred  way  to  protect  one  activity  from  another  when  they 
share  a single  CPU  is  to  use  a virtual  machine  operating  system.  Such  a system 
creates  an  artificial  machine  environment  for  each  user.  As  a result,  the 
operational  version  of  an  installation  OS  can  be  isolated  from  the  version 
used  by  developers.  This  is  particululy  helpful  when  the  OS  itself  is 
being  modified. 

A great  deal  of  the  benefit  from  a support  system  comes  from  its  library 
capediilities.  It  is  the  repository  of  all  the  project  data.  All  project 
personnel  can  be  given  access  to  project  data  to  learn  it,  to  use  it,  to  work 
on  it,  or  to  analyze  it  for  design  or  management  reasons.  Nothing  gets  lost 
in  a programmer's  desk.  "Private"  code  becomes  "public"  code.  All  the 
communications  links  needed  to  move  the  project  forward  can  be  provided 
through  the  support  system.  These  capabilities  appeau:  in  vaurious  degrees  in 
the  laurge  CLEAVCASTER  system  used  in  IBM  program  development  [60]  , the 
planned  Information  Automat  [61] , ISOOS  under  development  at  the  University 
of  Michigan  [62] , and  in  the  minicomputer  operating  system  from  Bell 
Laboratories,  UNIX  [63]. 

2.3.3  Applicability  of  Technical  Tools 

Development  amd  test  tools,  particularly  structured  programming,  are 
better  established  than  analysis/design  tools.  The  reasons  for  this  are 
baLsed  on  the  characteristics  of  the  program  development  life  cycle  discussed 
in  the  next  chapter.  Briefly,  the  most  obvious  place  to  look  for  the  source 
of  program  errors  or  cost  overruns  is  to  the  individual  programmers.  When 
this  was  done  in  the  late  1960s  it  became  immediately  apparent  that  the 
large  number  of  people  who  entered  the  field  since  1960  were  not  doing  basically 
"good"  programming.  In  the  '40a  and  '50s  programmers  caurried  a task  from 
stairt  to  finish  by  themselves.  They  did  structured  programming  and  stepwise 
refinement  without  thinking  adxsut  it.  It  was  the  natural  way  to  work. 

But,  because  they  had  not  thought  about  techniques,  they  did  not  document  or  teach 
the  methodology  for  "good"  programming.  In  the  '60s,  individual  jobs  were 
fewer  (except  for  the  special  case  of  personal  computing)  and  team  jobs  failed 
to  give  individuals  a context  in  which  to  develop  "good"  habits.  Techniques 
such  as  structiired  programming  vere  rediscovered  by  theoreticiauis  who, 
fortunately,  ej^lained  their  ideas  in  a teachadile  format.  Then,  when  the 
team  programmers  learned  how  to  distinguish  "good"  programs  from  "bad" 
ones,  the  largest  pairt  of  the  quality  amd  cost  problems  was  solved.  Or  so 
it  seemed.  In  fact,  the  progress  mads  with  IPT  served  to  highlight  the  fact 
that  good  programmers  cannot  compensate  for  poor  designs.  This  discovery 
led  to  the  development  of  improved  design  methodologies  which  could  interface 
with  and  take  aidvantage  of  IPT.  Still  there  were  serious  problems,  now  due 
to  the  frequent  changes  and  misunderstandings  aurising  from  poor  requirements 
statements.  So  in  the  '70s,  work  on  requirements  analysis  techniques 
accelerated. 

2. 3. 3.1  Scope  of  Selected  Tools 

Tools  developed  for  specific  purposes  tend  to  be  efficient  only  within 
the  limits  of  their  design  even  though  they  may  be  used  else%ahere.  By 
analogy,  PORTKAN  fits  well  in  mathematical  problems  but  is  clumsy  as  a 
coaDercial  data  processing  language.  REVS  which  was  built  to  help 
requirements  Mritars  is  not  promoted  as  a design  tool.  SADT  is  offered  to 


cover  both  analysis  and  design  but  does  not  claim  to  be  a programming  tool 
at  the  implementation  level.  HIPO  exhibits  a different  type  of  specialization. 

It  addresses  only  documentation  and  does  not  have  the  f\inctional  sophistication 
of  the  design  approaches.  HIPO,  however,  requires  pr2ictically  no  investment  by 
the  user.  Self-teaching  is  quite  feasible.  It  does  not  do  much  but  what  it  does 
is  important,  of  wide  use,  and  economical.  So  far,  simplicity  and  economy 
coupled  with  formal  rules  and  standard  procedxires  characterize  IPT  tools. 

Other  analysis/design/development/ test  tools  tend  to  be  more  sophisticated. 

They  give  the  user  freedom  of  action  to  express  his  own  style  in  the  "eurt” 
of  analysis  emd  design.  The  price  of  stich  flexibility  is  that  they  cost 
more  to  acquire  and  use  and  generally  require  a training  course  to  be  used 
properly. 

lioae  of  these  tools  is  necessarily  important  to  a programner  worJeinq 
totally  alone,  particularly , when  he  is  the  only  person  that  will  ever  use 
the  program,  the  tools  pay  off  beat  in  team  projects,  either  medium  size 
projects  (Fig.  2.13)  where  analysis  and  high-level  design  may  be  merged  or 
large  size  projects  (Fig.  2.14)  where  more  specialization  of  skills  is 
found.  Looking  at  the  scope  of  just  a few  tools  (Fig.  2.15) , you  could 
conclude  that  REVS  is.  of  greatest  interest  in  large  projects  where  esquires 
ments  anaiLysis  is  a special  skill  separate  from  design.  IPT  fits  all 
cases.  Further,  the  stress  on  good  documentation  and  control  techniques 
lets  all  the  tools  contribute  to  better  maintenance  and  other  follow-up 
activities. 

2. 3. 3. 2 Benoits  of  IPT 

Efforts  to  put  a price  tag  on  the  benefits  of  IPT  have  been  largely 
unsuccessful  because  of  the  lade  of  controlled  experiments.  It  is 
equally  risky  to  draw  conclusions  from  isolated  examples  where  productivity 
and/or  quality  improved  simultaneously  with  the  introduction  of  IPT.  Analysis 
of  such  cases  often  shows  other  factors  also  contributed  to  the  iaprovasient. 

The  other  factor  is  the  increase  in  the  average  level  of  experience 

of  the  staff  over  the  period  of  the  sasq^le.  For  instance,  a trend  observed 
in  United  States  data  for  the  period  1574-76  will  reflect  the  fact  that  there  was 
Uttls  hiring  or  es^playee  turnover  among  prograaoners  for  those  years.  Ths  same 
people  were  being  observed  for  three  years.  Even  though  there  is  no  proof 
that  experience  improves  productivity  and  quality  it  is  highly  probabls  that 
if  an  improvemant  is  ohssrvsd  it  is  as  much  due  to  experisnes  as  anything  else. 

The  meet  believable  data  would  have  to  come  from  very  large  programming  organization, 
(over  300  programers)  which  have  many  joba  in  process  and  have  converted  some 
but  not  all  to  IPT.  In  this  case,  the  numbers  would  be  large  enough  to  allow 
averaging  over  aggregates  of  individuals  and  jobs.  The  mix  of  characteristics 
of  ell  the  IPT  jobe  can  be  assumed  to  be  the  same  as  the  mix  for  all  the 
non- IPT  jobs.  Now,  if  a trand  is  obaarvad  in  production  rates,  error  rates, 
or  resource  utilization  rat  as  it  could  bs  attrlbutad  to  IPT. 

Thsra  is  no  such  claar  cut  axperimsntal  analysis  to  this  author's 
knowladga.  Very  good  analyses,  however,  ara  attributabla  to  Faigan  [51]  and 
Jones  [9] . Both  had  access  to  extanslve  production  and  maintananca 
racorda  on  numerous  program  systmns;  however,  the  data  that  had  been  collacted 
was  not  eoBplats  nor  was  it  consistantly  deflnad.  Thsrafora,  many  of  tha 
analytical  rasults  rapraaent  tha  investigator's  insight  and  consensus  opinions. 

Fagan  obaarvad  tha  programming  locations  of  IBM  in  Kingston  and  Poughkaepsia , 

New  Fork;  Jonas  obssrvad  a smallsr  group  at  IBM  San  Jose  but  augmented 
his  data  from  other  sources.  Fagan  predicts  that  structured  prograsm  undergoing 
design  and  code  inspections  will  have  substantially  fewer  errors  per  KLOC  than 
structured  programs  without  formal  inspections.  His  measuramenta  show  that 
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inspections  I , auid  I are  the  key  to  the  saving.  They  catch  errors  early  and 

avoid  error  propagation.  Both  inspections  occur  before  any  significant  machine 
time  is  used;  that  is,  before  program  unit  debugging  steurts.  Fagan  also 
finds  a 20%  saving  in  the  coding  portion  of  development.  Jones,  working 
from  a different  data  base  and  a planning  model,  compared  IFT  projects 
with  other  methods:  an  "old  style"  in  which  program  unit  debugging  was 
the  eaurliest  quality  control,  an  "OS/360  style"  in  which  design  verification 
was  also  done,  a "structured  style"  combining  design  verification  with 
top-down  design  and  structured  programming,  and  a "modem  style"  where 
inspections  supplement  the  structxired  style.  Jones  also  found  about  40% 
reduction  in  errors/KLOC  ^en  comparing  OS/360  style  to  modem  style. 

Savings  in  post-release  service  are  projected  by  Jones 

ranging  from  40-95%  of  the  four  year  cost  of  fixing  errors  in  program 

products  with  many  users.  Considering  that  in  four  years,  service  of  OS/360  style 

programs  uses  one-third  as  many  programmer-years  and  one-and-a-third  as  many 

computer  hours  as  the  total  initial  development  of  the  type  of  products  Jones 

is  concerned  with,  IFT  sav^jigs  represent  big  money. 

The  effects  of  fo\ir  different  qiulity  control  approaches  are  shown  in 
Table  2.3.  Using  Jones'  old  style  as  a base,  the  errors 
found  in  a project  are  allocated  to  the  development  cycle  stages:  design, 
code,  debug,  test,  emd  post-release.  Thus,  in  old  style  no  errors  are 
found  by  explicit  quality  controls  until  the  debugging  stage.  There,  15%  of 
the  errors  are  found;  60%  are  found  in  system  integration  and  test;  25%  of 
the  errors  remain  for  users  to  find.  By  contrast,  most  of  the  errors  are 
found  in  the  design  and  code  stages  when  the  Modem  Style  is  followed. 

The  table  entries  for  Old  Style  and  OS/360  Style  are  interpreted  from  Jones ' 
data.  The  Structured  Style  is  an  average  of  data  from  several  sources  where 
ZPT  was  used  but  inspections  were  not  used  because  the  project  team  preferred 
structured  walkthroughs  or  even  less  rigorous  reviews.  Modem  Style  is  an 
average  of  Jones,  Fagan,  and  other  sources  which  reflects  a range  of 
optimism  allocating  from  25-55%  of  the  errors  found  to  the  design  stage. 

An  important  feature  of  the  various  techniques  is  the  cascade  effect 
of  early  error  detection.  An  error  eliminated  during  design  tends  to  forestall 
additional  errors  later.  As  a result,  the  total  n\jmber  of  errors  encountered 
in  a modem  style  project  is  less  than  in  an  old  style  project.  Some  experts 
in  large  system  design  predict  that  IFT  eventually  will  be  ten  times  better 
than  old  style.  Here,  a more  modest  improvement  of  45%  is  projected  based 
on  what  has  been  achieved  to  date. 

It  is  possible  to  see  the  dramatic  value  of  IFT  by  using  the  costs  of 
fixing  an  error  detected  in  a given  stage  (Table  2.4)  to  estimate  the 
relative  costs  of  each  quality  control  approach.  Remember  that  the  cost  of 
an  error  is  proportional  not  only  to  the  difficulty  of  finding  a solution 
(fix)  but  also  to  the  nuinber  of  people  whose  progress  is  stalled  while 
waiting  for  the  fix.  During  design  there  are  only  a few  people  involved 
auid  they  ]cnow  their  product  intimately.  During  test  there  are  many  people 
involved  who,  at  best,  know  only  a small  part  of  the  product.  After  release, 
the  errors  are  reported  by  users  vrtio  submit  incorrect  and  duplicate  reports 
to  maintenance  teams  who  may  be  quite  unfamiliar  with  the  nuances  of  the 
product  design.  These  factors  account  for  the  huge  increase  in  cost  per  fix 
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I at  later  stages.  If  you  were  to  translate  the  relative  factors  to  r 

dollars,  you  would  iomediately  see  the  benefit  of  cleaning  up  errors  early, 
dsing  the  values  in  Tables  2.3  and  2.4,  Table  2.5  presents  the  relative  cost 
i per  stage  and  the  total  cost  per  project  of  quality  control.  The  details  are 

carried  out  to  decimal  values  to  be  consistent  with  Tables  2.3  and  2.4.  The 
totals  are  rounded  off  since  the  original  data,  while  taken  from  actual  environments, 
is  not  precise.  Still,  there  is  a 7:1  improvement  in  quality  projected  over 
the  period  1960-1976.  If  the  trend  is  consistently  maintained  the  vendor's 
costs  will  continue  to  go  down  but  the  main  beneficiary  will  be  the  user  who 
%rlll  have  fewer  problems.  For  every  25  errors  that  reached  the  customer 
under  the  Old  Style,  only  2-4  should  reach  him  today  amd  still  fewer  in  the 
future.  This  speaks  well  for  the  quality  control  aspect  of  program  development 
but  it  does  not  address  other  issues  of  productivity  and  cost. 

Using  the  same  data  as  Fagan  amd  Jones,  Home  in  early  1976  set  out  to  ^ 

show  the  effectiveness  of  IPT  in  terms  that  non-programming  managers  would 
understand.  This  objective  would  require  statistically  valid  results  large 
enou^  to  be  convincing.  This  objective  was  not  achieved  for  a variety  of 
reasons;  the  most  ioportant  of  which  was  that  the  source  data  had  not  been  obtained 
for  the  purpose  of  this  study.  Host,  of  the  data  was  used  by  local  managers 
to  track  departmental  perfissmance.  Home  needed  to  draw  general  parameters  out  of 
data  that  was  collected  for  local,  specialized  purposes.  She  had  eoUsctad 
a considerable  amoimt  of  data  on  IBM  system  control  programs  (e.g. , MTS,  VSl) 
and  ^plication  programs  (e.g.,  IMS,  health  industry  packages,  government 
and  commercial  contract  programs) . m this  list  were  very  large  camples 
programs,  widely  distributed  programs,  small  sizzle  programs,  one-user  programs. 

The  data  included  varying  degrees  of  detail  on  program  size,  size  of  changes 
to  existing  programs,  IPT  used,  prograsnaer-months,  estimatad  difficulty, 
productivity,  errors  found  before  and  after  release,  and  other  items. 

However,  the  data  was  uxureliable.  Measurements  were  not  consistent  and  were 
not  well  defined.  For  example,  one  program  appeared  in  four  reports  each  of 
%ihich  gave  different  size,  productivity,  cost,  and  error  rates  per  '.‘'IJOC. 

"Productivity'*  was  sometimes  to  lS2C/pragranBtar-month  and  in  other 

cases  cover sd  noc/project  persannel-months  bixt  it  was  hard  to  tell  which 
was  which.  The  IPT  usage  reports  were  often  subjective.  Tou  could  not 
tell  if  a project  reporting  ST  usage  understood  the  techniques  claimed. 

For  instance,  inspections  are  different  from  walkthroughs  but  few  people 
were  aware  of  the  difference.  The  applications  people  did  not  report  cost 
or  post-release  errors.  Rsports  for  nsw  programs  were  intsraixsd  with 
reports  for  nsw  releases  of  old  programs.  In  fact,  in  the  total  KLOC  in  the 
programs  surveyed  there  were  far  more  iSJOC  of  old  code  than  thera  were  now  or 
changed  lOJX.  And,  to  complicate  aattars,  the  individuals  who  collsctsd  ths 
data  were  no  longer  around  so  it  was  difficult  to  clarify  issues  of  dsfinition  I 

and  scope. 

Msvertheless , in  the  analysis  by  Home  and  this  author  it  was  possible 
to  confirm  the  belief  that  IPT  improves  the  program  development  process  based 
on  the  following  indications  (as  oppossd  to  proofs) : 

1.  For  projects  of  cos^arabls  difficulty,  IPT  increassd  prodtictlvlty 
in  KLOC/people-ysars  on  ths  avsrage. 

2.  Managers  were  willing  to  cosmiit  to  higher  productivity  on  new  projects 
incrsasing  the  KLOC/prograamer-month  used  in  their  estimating  tables. 
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3.  Managers  were  willing  to  commit  to  lower  objectives  in  terms  of 
errors/KLOC  in  post-release  code  because  they  had  confidence 
in  their  quality  control  technicpies. 

- 4.  Managurs  were  convinced  that  IPT  helped  them  and  they  were 
volunt^u:ily  adopting  the  techniques. 

5.  Progr^anmers  were  willing  to  adopt  new  techniques  in  1976  whereas 
they  shied  away  from  them  in  1974-75. 

On  the  negative  side,  the  indications  were: 

1.  It  is  so  far  not  possible  to  separate  the  benefits  of  one  technique  from 
emother. 

2.  Unusually  high  productivity  (found  in  a few  1-5  programmer 
projects)  is  due  more  to  outstanding  progriumners  than  to  IPT. 

3.  Determining  the  effect  of  IPT  (or  any  other  process  modification) 
on  quality  probably  requires  about  two  years  of  post-release  error 
data.  However,  when  trends  have  been  established  for  several  programs, 
it  should  be  possible  to  model  the  results  to  predict  quality  in 
other  programs  given  only  pre-release  error  data. 

4.  For  every  set  of  data  supporting  IPT,  there  is  a counterexample. 

5.  The  range  of  results  with  IPT  overlaps  the  range  without.  Since 
the  data  is  statistically  weaJc,  no  conclusion  can  be  drawn  about 
the  average  results  due  to  IPT. 

€.  If  IPT  data  solidifies  so  improvement  in  the  averages  can  be  seen, 
the  effect  on  process  measurements  may  remain  invisible  because  the 
savings  c^m  be  masJced  by  other  factors. 

o the  cost  of  system  support,  integration,  and  release  may 

dwarf  the  cost  of  new  code  so  IPT  benefits  appear  negligible. 

o the  savings  due  to  IPT  may  be  immediately  reinvested  in  additional 
progr2ua  functions;  i.e.,  total  outlay  is  not  reduced. 

o the  reduction  in  errors/KLOC  may  be  accompanied  by  am  increase  in 
KLOC  so  the  user  sees  no  change  in  practice. 

o the  savings  in  programmer-hours  may  simply  reduce  programmer 
overtime  without  changing  salaury  costs  or  programmer 
population . 

An  interesting  sidelight  of  the  Home  study  was  due  to  an  attempt  to  fit 
curves  to  the  data.  The  purpose  was  to  be  able  to  predict  costs  and  error 
rates  by  class  of  prograun,  difficulty,  and  worJcload.  The  form  of  the  curves  that 
best  fit  the  data  was 

k (A)  (A  A)  + a 
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wher*  A is  the  size  in  KLOC  (or  nodules)  of  the  program  systems, A A is  the 
number  of  new  or  changed  KLOC  (or  modules) , ]c  is  a factor  related  to 
programmer  productivity,  and  ^ is  a constant  related  to  design  quality  and 
process  management.  The  equation  reflects  the  discussion  of  Chapter  1 by 
representing  complexity  as  ^ and  viorkload  as  A A/A  emd  their  product  as 
(A)  (^  A) . Fitting  this  expression  to  error  data  yields  values  of  jc  and 
The  vauriable  portion,  can  be  inteurpreted  as  the  number  of  errors 
delivered  in  a program  system  due  to  pzragrammer  activity.  The  constant  portion, 

£,  which  should  be  zero  is  the  number  of  delivered  errors  due  to  the  development 
^oceaa.  The-  surpris*  was  that  in  this  model  idien  k • 0 (i.e,  each  programu: 
did  a p«]c£ect  job  of  program  uxtit  coding  and  debugging) , ^ was  non-zero.  The 
process  itself  was  introducing  a certain  number  of  defects.  If  the  analysis  is 
correct,  these  system  errors  must  be  due  to  (a)  inadequate  design,  (b)  errors 
put  in  after  unit  debugging,  and  (c)  the  nature  of  teat  and  release  activities. 

IPT  is  expected  to  improve  design  and  minimize  the  test  and  release  activities. 

Support  tools  that  hold  the  system  on-line  can  reduce  handling  errors  after 
unit  debugging.  Therefore,  the  new  technology  may  get  ^ down  to  zero  where 
it  belongs. 

A different  approach  to  evaluating  IPT  was  takes  by  Hunter  and  Reed  [64] 

\dio  subjectively  rated  the  use  of  IPT  on  a large  project  at  Barclays  Bank  Ltd.  in 
the  Kingdom.  They  found  significant  but  unquantified  gains  in  product 

quality,  enhanced  personnel  skills  and  motivation,  and  about  25%  better 
productivity  on  other  jobs  with  which  they  were  familiar.  The  techniques 

used  wholly  or  in  part  wares 

o structured  design 

o pseudo-code  program  design  language 

. o structured  progranaing 

a top-down  ia^loentation  and  testing 

o development  sigpport  library 

o librarians 

o walkthroughs 

o team  operations. 

The  project  received  advice  and  assistance  from  the  group  idio  tnrote  the  IPT 

[50] . AS  a result,  the  joint  Barclay-IBM  team  was  able  to  get  off  to 
a fast  start.  At  the  end  of  the  project,  a set  of  questions  was  prepared  to 
show  idiere  IPT  made  a difference.  The  strong  positive  bias  of  the  answers 
showed  that  IPT  had  been  a success  in  this  particular  project.  A sviomary 
of  the  questions  covering  manageability,  productivity,  maintainability,  1 

and  reliability  shows  how  the  qualitative  assessment  turned  out  [Table  2.6] . , I 


Th«  conclusions  of  this  study  parallel  those  given  earlier: 


o ZPT  provides  a framework  for  project  development  vdiich  is  better 
than  the  traditional  approach  amd  should  become  the  standard. 

o The  techniques  are  not  a 'miracle  formula’  for  a successful  project, 
without  a sensible  schedule,  strong  technical  leadership,  and 
competent  design  amd  programming,  a project  will  fail  with  amd 
without  IPT. 

o Ideally,  projects  should  introduce  most  of  the  techniques  together, 
staurting  as  eaurly  in  development  as  possible.  The  techniques  fit 
naturally  together  amd  in  some  cases  depend  for  their  success  on 
being  used  in  conjiinction  with  others. 

o However,  a subset  of  the  techniques  can  easily  be  used  as  a 
starting  point.  It  should  consist  of  structured  programming 
(including  pseudo-code) , use  of  a development  support  libraary 
approach,  and  walkthroughs. 

o Standards  in  many  cases  need  to  be  altered  to  accommodate 
development  along  IPT  lines. 

o The  overhead  cost  of  learning  the  techniques  is  almost  certainly 
recovered  inside  one  year. 

o IFT  is  not  a rigid  or  inflexible  set  of  rules.  Rather  it  is  a 

group  of  related  concepts  and  techniques.  It  is  natural  and  inevitable 
for  different  projects  to  use  and  interpret  IPT  someidiat  differently. 

Taking  advantage  of  the  plusses  of  IPT,  it  is  likely  that  modem  style 
program  development  will  lead  to  quality  and  productivity  improvements.  IPT 
may  be  hard  to  measure;  however,  the  improvements  due  to  IPT  over  time  will  be 
measurable.  By  1930,  IPT  should  generate  aUsout  40%  is^rovement  of  the  old  style 
Of  1970  [65].  That  is,  40%  fewer  errors  found  during  development  and  more  than  a 
40%  reduction  in  errors  in  code  released  to  users  (because  of  the  cumulative 
effect  of  early  discovery) . Project  resource  requirmnents  per  KLOC  should 
also  go  down  about  40%  represented  by  an  increase  in  KLOC/programmer-month 
and  in  KLOC/computer-hour.  The  range  of  1-4  KLOC/programmer-month  (before 
adjusting  for  system  sr^port,  system  test,  release,  and  maintenance  as  discussed 
in  Chapter  10)  is  achievable.  This  outlook  is  within  the  range  of  the  technology 
forecast  for  1985  made  by  the  Air  Force  Systems  Command  in  its  1972  study 
"Information  Processing/Data  Automation  In^licationa  of  Air  Force  Command 
and  Control  Requirements  in  the  1980a  (CCIP-85)"  reported  by  Boehm  and 
Kosy  [66] . Their  range  is  very  large  due  to  the  inherent  variability  of 
programming.  It  forecasts  prodxictivity  of  3.7  KLOC/programmer-month  in 
1980  rising  to  5.3  lOjOC/pm  in  1985  compared  to  1.4  KLOC/pm  in 
1970.^  The  increase  is  attributed  to  IPT  but  includes  programming  languages, 
design  aids,  and  reuse  of  common  functions  (macros/library  programs). 


These  numbers  are  the  midpoints  of  ranges  estimated  from  a logarithmic  scale 
in  Boehm.  Boehm's  chart  shows  0.4-7  KLOC/pm  in  1980,  0.6-10  KLOC/pra  in  1985, 
and  0.2-2. 6 KLOC/pm  in  1970.  Kosy  has  a similar  chart  with  a slightly  different 
scale  tdiich  yields  a more  optimistic  rauige  in  1980  of  1.4-15.5  KLOC/pm.  The 
lower  range  is  probably  more  realistic. 


511 


The  accompenying  text  is  part  of  a larger  chapter  covering  system 
characteristics  and  status  control  tools  as  well  as  svqpport  tools.  The 
entire  reference  list  is  included  hers  since  there  are  some  useful  items 
in  it. 
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Major  project  checkpoints 


Project  checkpoints 


Items  to  be  reviewed  via  a structured! 
walk-through  | 
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TABLE  2.3  QUALITY  CONTROL  - % OF  ERRORS  FOUND  BY  FOUR  QUALITY  CONTROL 
APPROACHES 
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PROJECT 

STAGE 


DESIGN 


CODE 


DEBUG 


QUALITY 

CONTROL 

APPROACH 


OLD  STYLE 


OS/ 360  STYLE 


(Cost  relative  to  OLD  STYLE 


TOTAL 

PROJECT 

(rounded 

value) 


37.5  1260.0  900.0  2200 


36.75  1029.0  529.2 


STRUCTURED  STYLE  4.35  5.8  21.75  426.3  104.4 


.MODERN  STYLE 


7.425  6.875  9.625  231.0  59.4 


SCD  I 


YES  DUE  ::o 

TO  IPT  NO  EFFECT 


MANAGEABILITY 


Are  major  schedules  being  net  more  ^ 
often?  ( r ) 

Are  budgets  being  net  more  often?  ( 

Are  milestones  and  checkpoints  ^ 

easier  to  set  up  and  to  enforce?  ( 

Are  intemediate  project  milestones 
and  checkpoints  being  met  more 
often?  ( 


Do  project  managers  get  better  status 
reports  to  track  progress?  ( i 


Is  the  developing  system  more 
visible,  centralized,  and  open  to 
inspection? 

/ 

Is  the  user  more  involved  with 
the  system,  development  effort? 

Ccn  changes  be  made  to  the 
developing  system  more  easily? 


( /) 
( /) 
( /) 


o Can  part  of  the /'system  be  exercised 

and  shown  to  the  user  befpre  the  end  ✓ 
of  the  project?  v ( 

o Are  project  managers  more  confident 

about  their  ability  to  manage  and  is  / 

their  job  satisfaction  higher?  ( 

o Are  analysts  and  programmers  more  / 
satisfied  with  their  jobs?  ( 

0 Can  people  be  phased  into  projects 

more  easily?  ( ) 

o ■ Are  clerical  tasks  easier  to 

delegate?  ( 

o Is  the  user  more  satisfied?  ( ✓I' 


Is  the  chief  executive  more 
satisfied? 

Is*  the  DP  manager  more  satisfied 


( ^ 
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( ) ( ) ( ) 

< 

( ) ( ) ( ) 


( O ( ) ( ) 


( ) ( ) 
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( /T  ( ) ( ) 


( '^  ( ) ( ) 

( ) ( ) 

( ) ( ) ( ) 


( ) ( ) ( *^) 


( ^ ( ) ( ) 

( *^)  ( ) ( ) 


( ^)  ( ) ( ) 
( ( ) ( ) 


TABLE  2.6  Subjective  Evaluation  of  IPT 


Sheet  1 
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VES  TO  IPT  NO  EFFECT 


PRODUCTIVITY 
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Is  there  less  confusion  between  , 
users  and  DP? 

( 
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( 

) 

( 

) 

0 

Is  the  Identification  of 
responsibilities  more  sharply 
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( 
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) 

( > 

0 

Can  analysts  and  programmers  find 
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( 1- 

( 

) 

( 

) 

0 

Can  programmers  schedule  their 
time  more  effectively? 
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) 
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drivers  than  similar  systems 
produced  In  the  past? 

( 

( 

) 
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0 

Is  there  less  throw-away  code? 
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( 

) 

( 

) 

o 

Is  the  amount  of  machine  test  time 
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) 
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Does  It  seem  that  DP  Is  producing 
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TABLE  2.6  Subjecclve  Evaluation  of  IPT  Sheet  2 
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Does  the  developing  system  seem  to 
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) 
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Does  it  take  less  time  and  effort 
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Is  integration  test  less  traumatic 
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Once  into  operation,  does  Che  system 
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Does  it  take  less  time  to  make  a 
maintenance  change? 
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Is  Che  documentation  more 
meaningful  on  this  sys'temY- 
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update?  ( *’’0 
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Do  programmers  show  less  reluctance 
to  maintain  someone  else's  coda?  ( 


Is  there  less  of  a stigma 
surrounding  maintenance? 

Are  fewer  programmers  required  to 
- -maintain  this  system  than  similar 
systems? 
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FIG.  2.13  ASSIGNMENTS  IN  MEDIUM  SIZE  PROJECTS 
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FIG.  2.14  ASSIGNMENTS  IN  LARGE  PROJECTS 
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ABSTRACT  ^ 


SADT  la  a language  for  describing  systems.  It  encourages  multiple  de- 
scriptions of  a system  from  different  viewpoints.  We  sketch  its  basic 
characteristics  stressing  the  natural  synergy  between  a language  and 
the  thought  patterns  it  Induces.  SADT  has  been  successfully  applied  to 
many  diverse  systems  to  perform  requirements  definition  and  functional 
analysis.  It  has  also  been  used  for  both  software  design  and  for  plan- 
ning in  non-computer  related  areas.  We  detail  actual  approaches  used 
in  applying  SADT  that  Integrally  involve  customer  analysts.  Some  repre- 
sentative examples  of  SADT  projects  are  discussed,  stressing  their  dis- 
tinctive features. 
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1.  WHAT  IS  SADT? 


SAOT  ® , Structured  Analysis  and  Design  Technique,  Is  a language  for 
describing  systems  together  with  a methodology  for  producing  such  de- 
scriptions. Here  a "system"  may  be  defined  as  any  combination  of  machin- 
ery (hardware),  data,  and  pesple  working  together  to  perform  a useful 
function.  More  generally  a system  can  be  viewed  as  consisting  of  things 
(objects,  documents,  or  data),  happenings  (activities  performed  by  people, 
machines,  software,  etc.),  and  their  Interrelationships.  This  fundamen- 
tal starting  point  accounts  for  SADT's  broad  applicability.  For  example, 
the  definition  of  requirements  (without  necessarily  specifying  whether 
they  are  to  be  achieved  by  people,  by  softwere,  by  machines),  a struc- 
tured software  design,  and  even  a p3  organisation  (people,  papers,  and 
procedures)  are  "systems"  and  thus  amenable  to  a precise  SADT  descrip- 
tion. 

SADT  was  developed  Initially  by  Douglas  T.  Ross  during  the  period  1969- 
1973.  It  was  drawn  from  his  own  ongoing  work  in  problem  solving  and  in 
software  engineering  dating  back  to  the  late  1950's.  It  was  strongly 
influenced  by  ideas  from  Hori’s  human-directed  activity  cell  model  [2], 
structured  programming  and  Its  supporting  procedures,  software  engineer- 
ing, general  systems  theory,  and  even  to  some  extent  cybernetics. 

SADT  has  been  further  developed  and  refined  at  SofTech  as  a result  of 
extensive  use  dating  from  1973.  It  has  thus  far  been  successfully  ap- 
plied to  a variety  of  complex  system  problems  ranging  from  real-time 
telephonic  coumunlcations  design  and  computer-aided  manufacturing  to  the 
analysis  of  funds  management  and  of  military  training. 

1.1  THE  NOTATION  AND  ITS  MODES  OF  THOUGHT 

An  SADT  system  description  Is  a set  of  diagrams  each  depicting  only  a 
limited  asiount  of  detail.  The  diagrams  expose  a system  structure  a piece 
at  a time  from  the  top  down.  Quite  literally  SADT  espouses  the  principle 
that  "to  divide  is  to  conquer",  provided  we  know  how  the  divided  pieces 
are  combined  to  constitute  the  whole.  Tills  proviso  applies  to  the  dia- 
gram contents  as  well  as  to  the  relationships  among  diagrams.  The  rules 
of  language  usage  encourage  and  even  enforce  unfolding  a system  descrip- 
tion in  Intellectually  manageable  Information  units.  At  all  times  the 
system  structure  and  hence  the  relationship  of  any  part  to  the  whole  re- 
mains graphically  visible. 

As  a language  SADT  has  a syntax,  a semantics,  and,  of  course,  an  associated 
pragmatics  of  use.  Its  syntactic  notation  has  an  elegant  simplicity  that 
has  been  introduced  In  several  publications  [4,  S,  6,  11].  Diagrams 
consist  principally  of  boxes  (representing  parts  of  a whole)  Intercon- 
nected by  arrows  (representing  Interfaces  between  the  parts)  each  suit- 
ably labelled  with  nouns  (for  things)  or  verb  phrases  (for  happenings). 

Each  box  on  a diagram  (representing,  say,  a system  activity)  can  be  fur- 
ther detailed  on  a separate  diagram  with  more  Interconnected  boxes  and 
arrow.  The  notation  requires  that  a box  and  the  corresponding  diagram 
<l«£alllns  It  represent  exactly  the  same  part  of  a system.  In  particular, 

[SADT  Is  a trademark  of  SofTech,  Inc. ] 
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i-  the  external  Interfaces  In  the  fora  cf  Inputs,  outputs,  and  controls 

aust  Batch  ("the  geometry  goes  together").  Thus,  the  dlagraas  In  an 
I SADT  systea  description  can  be  placed  directly  into  a tree-llks  hiar- 

^ archy.  This  information  is  encoded  by  an  Indexing  scheae  in  a natural 

fashion  so  that  the  appropriate  context  for  any  diagram  can  be  iaDcdl- 
ateiy  determined.  Figure  1 depicts  two  diagrams.  A box  on  the  first 
diagram  is  detailed  by  the  second  diagram. 


language  and  as  a methodology  directs  and  disciplines  the  analysis  of 
systems.  Crucial  concepts  are  naturally  expressible  and  thus  are  more 
easily  discerned  and  then  comnunlcated.  For  Instance,  system  feedback 
and  control  can  be  concisely  and  clearly  captured.  It  is  indeed  "a  good 
notation." 

As  with  any  language,  ease  of  expression  encourages  certain  patterns  of 
thought.  Within  SADT  one  Is  naturally  led  ^ the  notation  to  initially 
bound  the  context  of  consideration.  That  is,  the  Inputs,  the  outputs, 
and  the  controls  of  a system  under  study  must  first  be  determined.  Next, 
both  the  purpose  and  the  viewpoint  for  the  analysis  must  be  identified. 
This  serves  to  separate  concerns  by  focusing  the  effort  so  that  extra- 
neous matters  can  be  avoided.  For  example,  if  the  purpose  of  an  analy- 
sis is  to  obtain  a management  overview,  then  the  system  objects  and  ac- 
tivities will  be  naturally  examined  from  a suitable  level  of  abstraction. 
Obviously  the  analysis  would  proceed  differently  if  the  purpose  instead 
was  to  produce  a software  design.  Similarly  the  system  designer  view  and 
the  user  view  of  a software  system  are,  of  necessity,  quite  distinct 
(but  related) . 

1.2  MODELING  FROM  A VIEWPOINT  FOR  A PURPOSE 

An  SADT  model  contains  a set  of  diagrams  that  describe  a system  from  an 
identified  viewpoint  and  for  a particular  purpose.  A model  can  provide 
a context  for  other  relevant  material  such  as  explanatory  text,  support- 
ing documents,  and  forms.  Often  multiple  models  of  a system  from  dif- 
fering viewpoints  are  required  for  an  adequate  understanding.  Figure  2 
represents  several  SADT  models  by  triangles.  It  suggests  the  range  of 
models  that  might  be  appropriately  built  during  the  development  of  a 
system  involving  computers  and  people. 

SYSTEM  DEVELOPMENT  CYCLE 

CUMHINI  A A N»V. 

INvmoNMCM  /\  /\  INViHONMIM 

MOUIL  » fc  MOnil 


Figure  2:  Models  appropriate  to  System  Development  Cycle  Stages 
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The  range  extends  from  conception  through  Installation  and  even  Includes 
the  management  of  this  process.  The  scope  of  certain  key  models  should 
be  mentioned.  A Requirements  Model  crystallizes  criteria  the  new  sys- 
tem must  meet  In  order  to  satisfy  the  requestor's  needs  (along  with  those 
of  the  manager,  users,  etc.).  A Functional  Model  expresses  %rhat  func- 
tions the  new  system  must  perform  and  their  Implications.  A Design 
Model  depicts  a particular  design  that  meets  the  requirements  by  perform- 
ing the  specified  functions.  Actually  the  design  model  In  Figure  2 Is 
shown  as  an  Interconnected  group  of  models.  This  represents  SADT's  capa- 
bility of  Isolating  Important  design  decisions  for  separate  treatment  as 
models . 

Figure  3 shows  two  models  (l.e.,  system  descriptions  with  different  view- 
points) sharing  common  detail.  Consistency  can  be  checked  by  "tying"  the 
different  models  for  a system  together,  l.e.,  by  making  explicit  their 
Interrelationships . 


Figure  3:  Two  Interconnected  models  of  a system  from  different  viewpoints 
1.3  P3  - the  people,  THE  PAPERS,  AND  THE  PROCEDURES 

Producing  an  SADT  system  description  Is  a demanding  technical  project  re- 
quiring management,  expertise,  and  regular  reviews  to  assess  current  stat- 
us. Clearly  extensive  Interaction  with  the  customer  along  with  "Inter- 
viewing" appropriate  documents  is  a necessary  step  In  systestf  analysis. 
SADT  models  and  procedures  provide  a methodology  to  structure  this  mass 
of  Information.  Building  models  to  reflect  different  purposes  and  view- 
points effectively  sorts  and  relates  the  system  details. 

Drawing  Inspiration  from  the  recent  Innovative  organization  of  a software 
project  Into  a team  of  specialists,  SADT  has  defined  functions  for  roles 
identified  as  important  in  obtaining  a simple,  clear,  correct,  and  com- 
plete system  description.  Figure  4 lists  job  titles  and  associated  duties 
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Figure  4:  Personnel  Roles  for  SADT 

Much  could  be  said  about  Che  specific  procedures  for  Interaction  and  re- 
porting. Here  we  mention  only  the  crucial  cycle  of  authoring  diagrams 
Chat  then  receive  a critical  reading  with  written  conments.  These  read- 
er reactions  are  placed  directly  on  the  diagrams.  The  author  must  re- 
act In  writing  to  these  comments  and  eventually  revised  diagrams  are  Is- 
sued. Only  if  necessary  do  the  author  and  reader  meet  to  discuss  any 
differences.  This  cycle  is  repeated  as  often  as  required  to  evolve  a 
complete  and  correct  model.  If  Che  readership  Is  suitably  selected,  this 
process  effectively  ensures  Che  clarity  as  well  as  Che  accuracy  of  Che 
resulting  system  descriptions.  In  effect  it  institutionalizes  an  "ego- 
less" system  analysis  process. 

2.  APPLYING  SADT 

In  this  section  we  indicate  the  diversity  of  areas  to  which  SADT  has  been 
applied.  We  comment  on  some  common  characteristics.  The  actual  approaches 
to  applying  SADT  In  terms  of  personnel  roles  and  the  Interaction  with  Sof- 
Tech  are  sketched.  Then  some  specific  examples  of  SADT  projects  are  dis- 
cussed. 

2.1  AREAS  OF  APPLICATION 


The  most  important  characteristic  of  SADT  application  projects  has  been 
a need  to  communicate  results  among  people  with  different  Interests  and 
backgrounds  who  must  be  kept  Informed  during  the  development  of  s large 
manual  or  automated  system.  The  SADT  development  process  is  based  on  the 
fact  that  analysis  and  design  of  complex  systems  requires  coordination 
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of  the  creation,  nodificatlon,  and  verification  of  the  systea  description. 
No  single  person  can  coaprehend  coapletely  every  aspect  of  a coaplex  sys- 
tea within  the  tlae  Halts  usually  laposed.  Although  SADT  has  been  pro- 
fitably used  on  one-  or  two-person  projects,  a typical  project  Involves 
several  persons;  the  largest  project  to  date  has  Involved  fifteen  analysts 
slaultaneously  working  on  different  aspects  of  the  requlreaents  for  a 
large  financial  Inforaatlon  system.  Nearly  all  of  these  projects  have 
required  frequent  coaaunlcatlon  of  the  results  froa  the  analysts  pro- 
ducing a systea  description  to  experts  In  areas  iapacted  by  the  systea 
and  to  aanageaent  responsible  for  utilizing  the  systea. 

The  applications  to  which  SADT  has  been  applied  are  sufficiently  diverse 
that  at  first  glance  no  pattern  appears  to  exist  that  would  indicate  where 
It  might  be  most  useful.  The  largest  use  to  date  has  been  performing 
requirements  definition  and  functional  analysis  In  areas  where  computer 
systems  are  to  be  developed.  However,  SADT  has  a? so  been  used  to: 

e Describe  aerospace  manufacturing  operations  to  sufficient 
detail  to  provide  the  basis  of  a long-range  plan  for  com- 
puter-aided manufacturing. 

e Design  a data  base  system  to  maintain  and  modify  the  records 
of  over  five  million  business  establishments  participating 
in  Federal  Unemployment  Insurance  programs  In  sufficient  de- 
tail to  produce  module  design  specifications  using  high-level 
structured  PL/ I. 

• Describe  how  the  peace-time  Amy  provides  a combat-ready 
tank  system  In  order  to  Identify  the  training  required  and 
the  clrciBostances  In  which  training  must  be  carried  out  to 
keep  a weapon  system  ready. 

e Document  the  current  budgeting  and  accounting  operations  of 
a large  government  agency  In  order  to  isolate  the  areas  which 
could  benefit  the  most  from  Improved  Information  support. 

The  pattern  linking  most  SADT  applications  appears  to  be  the  difficulty 
of  Che  problem  to  be  solved.  All  the  problems  listed  above  were  very 
hard,  and  to  solve  them  required  the  analysts  and  designers  to  approach 
the  solution  in  a highly-structured  wsy  and  to  work  as  a team  with  effec- 
tive division  and  coordination  of  effort.  SADT  contributed  in  both  areas 
and  allowed  personnel  who  were  not  particularly  knowledgeable  about  many 
aspects  of  the  area  being  studied  to  synthesize  the  Informstlon  obtained 
from  many  experts  and  to  develop  a solution  whose  adequacy  and  quality 
was  assured  by  regular,  critical  review  within  the  development  team  and 
with  potential  users  and  non-technlcal  personnel. 

2.2  APPROACHES  TO  APPLICATION 

Since  a key  part  of  SADT  Is  a structured  way  of  thinking  about  large  and 
complex  problems.  It  Is  best  learned  through  application  under  the  super- 
vision of  a person  experienced  In  its  use.  Serving  an  "apprenticeship" 

Is  the  preferable  approach;  training  Is  not  possible  solely  through  class- 
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room  or  self-paced  Instruction.  The  establishment  of  the  review  process 
requires  both  management  Involvement  and  the  training  of  commenters  to 
read  SADT  models  and  to  doctascnt  their  suggestions  directly  onto  copies 
of  the  author's  diagrams.  Project  managers  must  be  coached  to  take  ad- 
vantage of  the  Increased  visibility  of  the  analysis  and  design  process 
In  order  to  better  evaluate  project  status  and  to  Improve  estimating, 
pacing,  and  resource  allocation. 

The  normal  approach  to  Innovating  SADT  In  an  organization  has  been  to 
conduct  a technology  transfer  program  for  the  personnel  assigned  to  a 
single  project  (typically  5-10  persons).  Following  a two-week  course 
in  the  basics  of  the  methodology,  the  project  team  begins  to  work  on 
Its  problem  "under  the  wing"  of  an  experienced  SADT  practitioner,  the 
monitor.  The  monitor  assists  and  advises  project  personnel  In  the  use 
and  application  of  SADT  and  teaches  seminars  on  advanced  topics  as  they 
become  relevant  to  the  work  of  the  project.  The  monitor  trains  comment- 
ers, ensures  the  review  cycle  Is  working  properly,  and  trains  the  SADT 
librarian.  In  addition  the  monitor  frequently  provides  consulting  to 
the  project  manager  on  the  most  effective  use  of  SADT  and  the  evolving 
models,  and  reviews  the  quality  of  the  on-golng  work. 

A three-month  monitoring  period  has  usually  been  adequate  to  effectively 
transfer  technical  aspects  of  the  methodology  and  to  develop  the  dis- 
ciplined team  approach  that  provides  a large  fraction  of  the  observed 
SADT  benefits.  It  Is  desirable  for  the  project  to  remain  In  contact  with 
the  monitor  by  telephone.  The  monitor  should  also  make  a one-day  visit 
every  two  or  three  %/eeks  In  order  to  review  quality  and  to  help  the  pro- 
ject personnel  stand  back  and  take  a better  look  at  thalr  prdblems.  This 
approach  has  been  used  by  Softeeh  in  over  twenty  technology  transfer  pro- 
grams In  the  last  three  years. 

An  alternative  to  the  monitoring  approach  when  more  trained  SADT  person- 
nel are  available  Is  to  use  a combination  of  trained  personnel  and  learn- 
ers on  the  project,  with  the  most  experienced  personnel  snnltoring  the 
work  of  less  experienced  people.  However,  a monitor  who  Is  not  directly 
Involved  In  doing  analysis  or  design  can  more  effectively  review  quality 
and  provide  consulting  to  the  project  manager.  An  example  of  using  ex- 
perienced and  Inexperienced  personnel  together  Is  a project  to  study 
current  operations  and  to  develop  the  requirements  for  a financial  In- 
formation system  where  five  SofTcch  experienced  SADT  authors  are  working 
closely  with  ten  client  analysts  both  to  complete  the  project  and  to  train 
the  client  personnel  In  SADT. 

SADT  also  facilitates  analysis  and/or  design  of  a system  by  en  outside 
organization  In  order  to  fulfill  user  requirements.  For  such  a working 
relationship  to  be  effective,  continuous  end  effective  coemunlcetion  is 
required  throughout  the  project  since  the  analysts  and  designers  assigned 
to  the  project  are  not  likely  to  be  particularly  knowledgeable  about  the 
users'  areas  of  expertise.  SofTcch's  experience  has  been  that  an  evolv- 
ing SADT  model  with  Its  supporting  text  and  referencea  to  other  documenta- 
tion provides  the  understandable,  complete,  uniform  end  current  decuman te- 
tlon  such  a working  relationship  requires.  The  documentation  must  show 
analysis  or  design  decisions  In  context  so  that  they  can  be  chellamged 
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while  alternative  approaches  are  still  viable.  Such  documentation  must 
provide  the  reasons  for  decisions  and  facilitate  review  by  user  experts. 
Attention  must  be  given  early  to  establishing  the  review  cycle  and  to 
defining  how  analysis  and  design  decisions  are  to  be  critiqued.  In  all 
cases  where  these  steps  were  taken  SofTech  has  had  excellent  results  In 
solving  a wide  variety  of  user  problems  using  professional  analysts  whose 
generic  analysis  skills  are  enhanced  by  SADT. 

In  the  following  section  examples  of  the  three  approaches  described  above 
are  given  In  the  context  of  several  recent  applications  of  SADT. 

2.3  EXAMPLES  OF  APPLICATION 

This  subsection  describes  some  recent  SADT  projects  concerned  with  soft- 
ware design,  current  financial  operations,  computer-aided  manufacturing, 
and  training  of  personnel.  They  are  representative  of  the  spectrum  of 
areas  In  which  SADT  has  been  successfully  applied  to  large  systems  prob- 
lesw.  For  each  project  Its  goal  as  well  as  the  actual  vmrklng  relation- 
ship with  the  customer  are  outlined.  The  project  output  In  terms  of 
models  and  number  of  diagrams  are  also  presented.  In  some  cases  propri- 
etary rights  permit  only  a generic  description  of  the  project  and  the 
contracting  customer. 

2.3.1  PABX  Software  Design 

This  completed  SADT  project  was  to  produce  detailed  software  design  for 
a microprocessor-based  private  automatic  branch  exchange  (PABX)  telephone 
system.  It  had  to  acconnodate  a wide  range  of  Installation  size  (100 
lines  up  to  10,000  lines)  and  over  275  special  features.  The  functional 
specifications  were  given  using  a program  design  language  specially  tai- 
lored for  specifying  switching  programs. 

Three  customer  authors  and  one  SofTech  author  built  two  principal  SADT 
models  — a design  model  and  an  lisplementatlon  ondel.  In  all  about  eight 
models  and  approximately  500  diagrams  were  fashioned.  The  extreme  quan- 
tity of  diagrams  can  be  attributed  to  the  detail  to  which  the  analysis  was 
taken.  Boxes  at  the  bottom  level  sometimes  reflected  Individual  state- 
ments In  a PL/I-llke  language. 

In  many  Instances  the  diagrams  were  used  as  programmer  work  assignments. 
The  use  of  SADT  coupled  with  structured  programming  practices  led  to  an 
Increase  of  between  2 end  3 times  the  productivity  previously  attained 
by  the  customer's  prograsMrs.  An  outside  contractor  was  slso  used  for 
system  implementation  and  was  able  to  easily  read  the  diagrams  In  order 
to  produce  code. 

The  building  of  multiple  ondels  from  different  viewpoints  (e.g.,  operator, 
designer,  user,  and  manager  viewpoints)  was  a major  factor  In  this  highly 
successful  softwsre  design  effort.  It  served  to  Isolate  the  important 
system  Interfaces  thst  were  then  sccommodated  by  the  design.  The  central 
role  of  SADT  dlagrasw  as  self-contained  cosanmlcatlon  doctsments  was  re- 
peatedly underscored. 
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2.3.2  Financial  ManaReaent 


This  SADT  project  Is  describing  the  current  operations  of  a large  federal 
government  agency's  financial  management  system  for  the  purpose  of  Identi- 
fying needs  and  deficiencies.  After  construction  of  a current  operations 
model,  a requirements  model  for  a financial  management  infontatlon  sys- 
tem is  to  be  built.  As  many  as  15  authors  have  been  working  simultane- 
ously at  four  geographically  dispersed  sites.  Customer  participation  in 
the  modeling  effort  is  substantial  with  a 3 to  1 ration  of  its  authors  to 
Soflech  authors. 

Three  basic  models  are  being  constructed  (with  other  supporting  or  auxili> 
ary  models  as  needed):  Obtain  Financial  Resources,  Manage  Funding,  and 
Accounting.  About  150  diagrams  plus  many  pertinent  docurcnts  and  forms 
will  comprise  the  final  financial  managesient  system  description.  The 
eventual  goal  is  to  design  and  then  implement  management  information  sys- 
tems that  are  coordinated  and  Integrated  In  order  to  handle  the  antici- 
pated dynamic  growth  In  the  agency's  financial  Information  requirements. 
Design  models  may  be  built  to  realize  this  goal. 

The  PSL/PSA  system  (9,  10]  Is  being  used  to  record  the  activity,  data, 
hierarchy,  and  structure  expressed  in  the  SADT  models.  PSA  produces  re- 
ports that  suamarlze  this  Information.  The  system  detects  nasd.ng  con- 
flicts (as  a consistency  check)  and  also  guards  against  proliferating 
synonyms. 

Periodic  model  walk-throughs  using  project  personnel  were  a crucial  com- 
plement to  the  machine  checking  and  tracking  of  data  usage.  Judiciously 
invoked,  the  walk-throughs  have  proven  to  be  a major  device  to  keep  the 
project  manager  cognizant  of  the  current  status  while  exercising  the  dy- 
namic sequencing  Inherent  in  the  models. 

Several  facets  make  this  SADT  project  distinctive:  the  number  of  authors 
Involved  at  multiple  sites,  the  fact  that  they  are  preponderately  custom- 
er analysts,  and  the  use  of  PSL/PSA  as  a computer  support  tool  for  the 
SADT  methodology. 

2.3.3  Computer-Aided  Manufacturing 

As  part  of  the  U,  S.  Air  Force  Integrated  Computer-Aided  Manufacturing 
(ICAM)  program  SofTech  Is  using  SADT  to  develop  a manufacturing  "archi- 
tecture" — a generally  applicable  model  of  the  batch  manufacturing  pro- 
cess, with  Initial  emphasis  upon  sheet  metal  fabrication  and  subassei^ly. 
The  SADT  Architectural  Model  will  show  the  hierarchical  structure  of  all 
aspects  Including  machines,  materials,  processors,  procedures,  people,  and 
organizations.  It  will  describe  the  essentials  of  batch  manufacturing  that 
arc  common  across  companies  In  order  to  plan  subsequent  steps  of  the  ICAM 
program. 

The  project  will  utilize  SADT  to  capture  the  manufacturing  experience  of 
the  aerospace  and  Industrial  concerns  pertlcipatlng  in  the  project  (vis., 
Boeing,  Ceneral  Dynamics,  Gnassan,  Hughes  Aircraft,  Lockheed,  Rockwell 
International,  Onlted  Technologies  Corporation,  Vought,  Caterpillar 
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Tractor,  Clncinnatl-Milacron,  Glddlngs  and  Levis,  and  Keaney  and  Track- 
er). Hulciple  aodels  of  different  aspects  of  manufacturing  are  being 
developed  by  SofTech  analysts  based  both  on  interviews  with  experts  at 
the  participating  companies  and  on  documents  provided  by  them.  Extensive 
review  of  the  evolving  models  will  be  performed  at  each  company  by  com- 
pany experts  especially  trained  to  serve  as  the  Interface  between  their 
manufacturing  experts  and  the  SofTech  SADT  authors. 

The  PSL/PSA  system  [9,  10]  Is  being  used  to  store  the  evolving  models 
and  to  perform  a variety  of  consistency  audits.  Computer  support  to 
SADT  was  deemed  necessary  since  many  diagrams  %rlll  be  created  in  the 
multiple  models  representing  manufacturing  from  different  viewpoints. 

SofTech  Is  also  conducting  major  manufacturing  studies  In  the  areas  of 
advanced  processing  planning  and  geometric  modeling  for  Computer-Aided 
Manufacturing-International.  In  both  programs,  large  manufacturers  are 
working  closely  with  SofTech  using  SADT  models  as  the  main  communication 
vehicle. 

2.3.4  Army  Training 

SADT  Is  being  used  by  SofTech  In  conjunction  with  the  Headquarters  of 
the  U.  S.  Army  Training  and  Doctrine  Command  (TRAOOC)  to: 

• Identify  changes  In  Army  training  required  to  significantly 
Increase  combat  effectiveness. 

e Describe  how  Amy  training,  testing,  and  evaluation  programs 
will  operate  after  the  proposed  changes  are  accomplished. 

e Plan  how  the  changes  In  Army  training  will  be  undertaken  and 
how  progress  will  be  monitored  and  evaluated. 

The  project  Is  being  Jointly  funded  by  TRADOC  and  the  Defense  Advanced 
Research  Projects  Agency  (DARPA)  In  order  to  demonstrate  the  use  of  SADT 
In  the  context  of  the  Army's  current  effort  to  make  significant  Improve- 
ments In  training.  An  Important  project  objective  Is  to  assess  whether 
better  systems  analysis  methods  will  facilitate  the  definition  of  military 
training  as  a large  system.  SADT  models  are  being  developed  to  show  the 
Army  training  system  In  terms  of  clearly  described  activities  and  their 
Interfaces.  Such  a system  definition  should  provide  the  basis  for  more 
effectively  solving  many  small  problems,  which  must  be  handled  with  limit- 
ed resources  and  under  tight  deadlines.  In  the  context  of  overall  military 
objectives. 

The  project  Includes  an  Independent  evaluation  by  a team  from  the  Uni- 
versity of  Texas  under  the  direction  of  Or.  Cary  Borlch  which  will  pro- 
vide DARPA  with: 

e An  assessment  of  the  utility  and  effectiveness  of  SADT  la 
the  training  command  environment. 

a An  evaluation  of  the  Impact  of  the  project  on  TRAOOC's  capa- 
bilities to  define  and  manage  Innovations  la  Amy  training. 
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A recently  completed  project  task,  has  been  Identifying  changes  in  train- 
ing support  to  increase  the  combat  readiness  of  major  Army  treapons  sys- 
tems. The  tank  was  selected  as  a prototype  for  purposes  of  identifying 
the  training  support  needed.  An  SADT  modal  describing  the  tank  as  a 
total  weapon  system  [8]  was  used  to  determine  requisite  training  changes. 
The  model  consists  of  38  diagrams  with  supporting  text.  It  was  constructed 
by  a team  of  two  SofTech  authors  and  two  Army  authors  over  a period  of  four 
months. 

The  process  of  applying  SADT  included  interviewing  many  Army  experts 
and  obtaining  comments  from  25  Army  commentators  at  TRADOC  Headquarters 
and  at  the  Armor  School,  Fort  Knox. 

This  project  Is  different  from  thode  described  above  in  that  project 
goal  is  not  the  definition  or  tW  development  of  a computer-based  system, 
but  the  demonstration  of  how  SADT  can  be  used  for  analysis  and  planning  in 
non-computer  related  areas. 

3.  THE  SIGNIFICANCE  OF  SADT 

We  take  the  applicability  of  SADT  to  large  system  problems  as  an  estab- 
lished fact.  As  a language  for  describing  systems,  SADT  solves  the  simul- 
taneous need  for  precision  and  for  clarity  of  communication.  Its  notation 
Is  rigorous  without  being  unduly  formal,  for  a system  description  must 
encompass  myriad  technical  details  while  being  accessible  to  a reader- 
ship  concerned  principally  with  an  overview.  The  use  of  multiple  descrip- 
tions each  reflecting  a particular  purpose  and  viewpoint  represents  a 
major  advance  in  organizing  our  understanding  of  systems. 

SADT  may  point  the  way  to  the  eventual  Impact  that  language  theory,  and 
in  particular  semantics,  will  have  on  large  real  world  systems.  In  a 
sense,  SADT  provides  a semantic  framework  in  terms  of  which  all  analyses 
and  descriptions  are  done.  The  variety  of  successful  applications  serves 
as  a pragmatic  existence  proof  for  the  adequacy  of  the  SADT  framework. 

The  Reference  section  reflects  the  relatively  recent  emergence  of  SADT- 
related  literature  In  the  public  domain.  We  close  by  quoting  Dr.  Michael 
Mellch  (3]  of  the  Center  for  Naval  Analysis  on  using  SADT: 

"With  ay  Interests  In  performance  measurement  of  command  and  control  sys- 
tems, the  ability  to  partition  functions  unambiguously  gives  me  a big 
start  on  the  development  of  performance  measures.  I now  have  a way  to 
get  people  to  agree  about  what  they  think  one  of  these  systeas  is  supposed 
to  do.  Once  they  agree,  I have  a chance  to  decide  if  the  systems  actually 
do  what  they  are  supposed  to  do.  The  last  and  probably  one  of  the  most 
surprising  consequences  of  this  use  of  Structured  Analysis  has  been  the 
rs^J***’  severe  reduction  in  time  wasted  needlessly  arguing.  It  seems  that 
we  actually  can  define  and  stick  to  the  point  without  useless  wandering 
on  'sea  stories'  or  ' 1 remember  whens'." 
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DOMONIC  - A SYSTEM  TO  DOCUMENT,  MONITOR,  AND  CONTROL 


SOFTWARE  DEVELOPMENT 

SOFTI^ARE  LIFE  CYCLE  MANAGEMENT  (SLCM)  WORKSHOP 

Sponsored  by 

U.S.  Army  Computer  Systems  Command 
Warrenton,  Virginia 


INTRODUCTION 


During  1972,  Texas  A5^  University  made  a study  of 
docximentation  aids  for  NASA  at  Goddard  Space  Flight  Center. 
The  results  of  this  study  led  to  the  design  and  development 
of  DOMONIC  (A  System  to  Docianent,  Monitor  and  Control 
Software  Development) . From  the  early  stages  of  DOMONIC 
development,  technical  supervision  and  design  insight  and 
suggestions  have  been  supplied  by  Evminious  Damon  at  NASA 
Goddard  Space  Flight  Center. 

DOMONIC  is  written  in  COBOL  which  makes  it  hardware 
independent.  The  initial  version  was  developed  on  IBM 
360/370  systems  and  can  be  adapted  to  run  on  CDC  and  Univac 
systems.  When  operating  system  modifications  are  made,  an 
excessive  amount  of  program  maintenance  is  reqxiired  during 
later  stages  of  operation.  Since  each  manufacturer  has 
developed  their  own  operating  system,  DOMONIC  has  been 
written  to  be  operating  system  independent.  DOMONIC  can  be 
used  to  develop  software  for  any  machine  written  in  any 
language.  Software  for  minis,  micros,  or  large  computers 
written  in  high-level  languages  or  assembler  langiiages  can 
be  developed.  DOMONIC  can  be  operated  in  either  a batch  mode 
or  an  interactive  mode.  Both  versions  have  been  designed 
to  be  simple  to  use  by  all  people  involved  in  the  development 
process.  Every  effort  has  been  made  to  minimize  restrictions 


on  prograssners  who  use  the  system.  While  DOMONIC  is  very 
\}se£ul  in  all  phases  of  software  development,  the  existing 
software  systems  can  be  retrofitted  into  the  DOMONIC  data 
bases  allowing  the  systems  to  be  tested,  documented  and 
maintained  using  DOMONIC. 

To  better  understand  how  DOMONIC  works,  let  us  look  at 
the  block  diagram  of  the  development  process  (see  Figure  1) . 
Software  and  development  is  broken  down  into  development 
phase  and  maintenance  phase.  During  development,  systems 
must  be  specified,  designed,  programmed,  and  validated. 

Data,  during  all  these  phases,  is  deposited  into  a DOMONIC 
data  base.  Every'  action  of  all  participants  in  the  develelopment 
process  can  be  monitored  through  the  DOMONIC  monitor.  Docimients 
for  the  user,  operator,  maintainer,  and  program  librarian 
are  automatically  validated  by  DOMONIC.  Once  data  is  inserted 
into  the  data  base  by  anyone  during  the  development  process , 
this  data  can  be  X2sed  by  others  later  in  the  process.  For 
example,  the  system  specifier  may  input  such  information  as 
titles,  abstracts,  systems  specifications,  testing  criteria, 
block  diagrams,  etc.  The  system  designer  can  benefit  from 
any  information  created  by  the  systems  specifier.  The 
designer  may  add  additional  information  such  as  system 
flowcharts,  system  description,  data  description,  program 
standards,  program  design  and  test  proced\ires,  development 
schedules,  etc.  The  programmer  also  benefits  from  the  data 
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stores  as  a documentation  unit  within  DOMONIC.  The  programmer, 
after  entering  source  code  into  the  system,  can  automatically 
generate  program  flowcharts,  cross  references,  overlay  struc- 
t\ires,  glossaries,  etc.  The  desirable  extension  to  DOMONIC 
would  be  to  expand  the  system  to  automatically  produce  data 
layouts.  The  validator  also  benefits  from  previously  stored 
information.  Off-the-shelf  software  packages  to  validate 
and  test  software  systems  can  be  called  in  during  the  validation 
stage.  Error  and  test  data  gathered  during  the  development 
process  and  during  the  validation  stage  can  be  used  to  drive 
software  reliability  models. 

DOMONIC  can  be  used  to  assist  the  docxjmentation  process. 
D0M3NIC  can  be  used  as  a central  depository  of  all  information 
created  for  a software  system.  Information  is  only  entered 
once  which  results  in  the  elimination  of  redundant  information 
storage.  For  information  stored  in  the  data  base,  the  TIDY 
program  flowcharters,  text  editors,  and  other  documentation 
aids  can  be  directly  driven  by  D0(ft3NIC.  Cotq) laxities  of  job 
control  language  unique  to  individiial  docxxmentation  aids  will 
be  transparent  to  the  user.  New  documentation  aids  can  be 
easily  added  to  DOMONIC.  The  major  advantage  of  DOMONIC  is  that  a 
manager  can  quantitatively  specify  and  control  the  programming 
practices  and  standards  that  he  would  like  to  be  followed 
during  software  development  and  maintenance. 
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The  conceptual  block  diagram  of  the  DOMONIC  system  is 
shown  in  Figure  2.  When  a project  is  initiated,  through  the 
concept  of  data  base  template,  a manager  can  express  all  data 
that  is  to  be  collected  and  stored  in  the  data  base.  Through 
the  template  concept  and  by  using  a sophisticated  security 
system,  the  manager  can  classify  the  procedures  and  standards 
that  should  be  followed  in  software  development  and  can  later 
monitor  and  control  the  development.  The  word  "manual”  in 
this  figure  refers  to  the  manxial  produced  automatically  as  a 
result  of  the  documentation  process.  The  items  that  go  into 
a manual  are  specified  through  a recipe.  The  recipe  indicates 
what  information  will  be  taken  from  the  data  base  and  used 
to  drive  documentation  aids  to  produce  a manual  for  docxmien- 
tation.  The  same  concept  can  be  used  to  drive  software  testing 
and  validation  packages . 

Docxnnentation  that  can  be  produced  by  such  a system  as 
DOMONIC  can  be  broken  down  into  system,  global,  and  local 
documentation  (see  Figure  3) . A great  deal  of  off-the-shelf 
packages  exist  to  produce  local  documentation.  Local  docijmen- 
tation  consists  of  listings,  flowcharts,  data  layouts,  cross 
references,  glossaries  and  text.  Similar  types  of  information 
should  also  be  generated  for  the  global  and  system  levels. 
DOMONIC  can  drive  such  aids  as  text  editors  and  flowcharters. 
Special  aids  have  been  developed  to  show  subroutine  connec- 
tivity, cross  reference  information,  overlay  maps,  subroutine 
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paramecers,  summaries,  etc.  Other  aids  can  automatically  tidy 
programs.  The  system  can  drive  compilers,  linkage  editors, 
loaders  and  utilities  which  produce  information  used  for 
docvnnentation.  A text  editor  that  could  be  used  on  an  IBM 
system  is  Text  360.  DOMONIC  can  drive  other  editors  when 
xised  on  Uni  vac  and  CDC  systems . Sophisticated  subroutine 
conhectivity  maps  can  be  drawn  which  are  similar  to  the 
hlerarchial  diagrams  vised  as  a part  of  HIPO  charts.  Special 
documentation  aids  to  produce  exhaustive  cross  reference 
Information  for  FORTRAN,  COBOL,  PL/1,  and  assembler  language 
programs  have  been  produced.  A TIDY  program  can  be  used  to 
automatically  renumber  statements , insert  spaces  for  clearer 
reading,  and  indent  sections  of  segments  relating  to  the 
same  control  sections. 

Overlay  maps  and  a detailed  svonmary  of  all  parameters 
can  be  automatically  generated.  Compilers , linkage  editors , 
loaders  and  utilities  generate  a considerab!*  e amovmt  of 
information  that  can  be  used  to  document  a program. 

Although  there  are  a number  of  automatic  flowchart  systems 
available,  these  systems  cost  a significant  amount  of  money 
each  time  they  are  installed  on  a new  computer.  An  advanced 
flowchart  system  has  been  developed  for  use  in  DOMONIC.  The 
system  can  draw  any  size  flowchart  symbol.  The  symbol  can 
be  any  shape  and  can  be  coTq)actly  placed  on  any  size  page. 


The  system  has  been  desigjied  to  be  completely  language 
Independent,  but  the  system  has  been  initially  implemented 
to  draw  flowcharts  of  FORTRAN  programs.  The  initial  version 
of  the  system  can  draw  detailed  flowcharts  or  flowcharts 
with  details  suppressed.  The  sy^'tem  has  been  implemented 
using  COBOL. 

A flowchart  has  been  drawn  of  the  simple  FORTRAN  program 
shown  in  Figure  4.  Notice  statement  three  has  a READ  state- 
ment  that  can  transfer  to  three  different  places.  Statement 
five  is  a logical  IF  statement  and  statement  seven  contains 
a DO  loop.  A detailed  flowchart  of  the  program  in  Figure  4 
is  shown  in  Figure  5.  The  entry  symbol  that  says  TAM  was 
created  to  show  the  versatility  of  symbol  shape.  Normally, 
the  symbol  would  be  an  entry  symbol. 

Notice  that  the  logical  IF  statement  is  created  by  using 
two  s3nnbols . The  DO  loop  has  been  created  by  using  four 
symbols.  Figure  6 shows  a details  suppressed  flowchart. 

The  READ  statement  has  been  compressed  down  to  a single 
input/single  output  symbol  emd  the  logical  IF  is  contained 
in  a single  diamond.  The  DO  statement  has  been  compressed 
to  a single  symbol. 
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MONITOR 

DOMONIC  is  a transaction  oriented  system  which  allows 
every  transaction  within  the  system  to  be  monitored.  Every 
activity  performed  by  anyone  using  this  system  can  be  recorded. 

A monitor  can  record  all  error  correction  data.  Data  gathered 
by  the  monitor  can  be  used  to  drive  reliability  models  and 
programmer  productivity  models.  The  monitor  makes  software 
development  projects  visible  to  management.  Once  a manager 
knows  the  exact  status  of  a project  then  he  can  control  software 
systems  development  through  DOMONIC. 

CONTROL 

DOMONIC  contains  a hierarchial  security  system  which 
allows  managers  to  dictate  the  level  of  access  to  software 
development  by  anyone  working  on  a project.  He  can  specify 
access  in  a hierarchial  manner,  where  he  allows  access  to 
data  by  the  people  directly  below  him  and  the  people  on  the 
next  lower  level  can  specify  the  modes  of  access  to  data  by 
the  people  below  them.  The  system  can  limit  access  to  data, 
‘iystem  commands,  output  generator  features  and  monitor  data. 
Through  these  systems,  the  manager  can  control  who  can  modify 
modules , what  can  be  done  to  the  modules , when  modification 
can  be  made  and  when  change  levels  are  frozen.  The  DOMONIC 
system  can  be  used  to  improve  chief  programmer  teams.  Teams 
are  normally  made  up  of  a chief  programmer,  a backup  programmer, 
additional  programmers  and  a programming  secretary.  The 
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program  secretary  maintains  the  program  production  libraiTr, 
collects  project  notebooks  from  programmers , makes  corrections 
and  executes  project  programs,  enters  new  output  and  source 
listings  in  project  notebooks  and  returns  project  notebooks 
to  project  programmers.  The  key  individtial  to  the  success 
of  the  chief  programmer  team  is  a programming  secretary. 

All  functions  of  this  individual  can  be  performed  automatically 
by  the  DOMONIC  system,  thus  reducing  the  manpower  necessary 

to  form  a chief  programmer  team.  ; 

1 

RELIABILITY  MODELS 

For  this  part  of  the  DOMONIC  development,  a Completion 
Model  and  an  Acceptance  Model  have  been  developed.  A 
Completion  Model  describes  the  status  of  the  program  under 

rl 

development.  An  Acceptance  Model  describes  the  reliability 
of  a completed  software  project.  Let  vis  look  first  at  the 
Completion  Model.  Activity  history  of  jobs  is  collected, 
as  shown  in  Figure  7.  During  initial  processing  a personal 
history  file  can  be  built  for  each  programmer.  During  later 
processing,  as  the  programmer  uses  the  system,  an  exact  comple- 
tion level  estimate  is  used  for  a system.  The  completion 
model  takes  advantage  of  the  fact  that  different  activities 
are  normally  performed  during”  different  stages  of  development  ’ ’ 

(see  Figure  8).  For  example,  during  early  development, 
sovirce  code  is  frequently  added,  some  of  the  source  code  is 
modified,  no  execution  is  performed  and  there  is  no  need  to 
remember  programs  and  produce  other  cosmetic  edits.  This  I 
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profile  is  different  for  the  mid- development  phase  and  the  >< 

final  testing  and  documentation  phases.  For  example,  in  the 
final  phase,  source  code  is  seldom  added  and  the  source  code 
that  is  there  is  seldom  modified.  Programs  are  frequentl/  < 

executed  and  docxmientation  aids  to  renumber  statements  and 
form  other  cosmetic  edit  are  xised.  The  personal  history  for 
each  activity  of  a programmer  is  created.  From  this,  profile 
completion  can  be  estimated  for  each  module.  For  example, 
completion  of  a single  module  is  demonstrated  in  Figtire  9 
which  plots  estimated  completion  level  versus  actual  completion 
level.  A composite  of  the  completion  level  for  all  of  the 
modules  can  be  plotted  versus  time  to  estimate  the  completion 
of  a total  project  (see  Figure  10). 


In  contrast  to  the  Cos^letion  Model,  which  can  estimate 
how  close  to  completion  the  system  is,  the  Acceptance  Model 
can  be  used  to  determine  the  reliability  of  a finished  product. 
This  model  uses  conventional  reliability  meastirement  technology. 
In  development  of  the  acceptance  model,  emphasis  has  been  placed 
on  making  the  technology  understandable.  Its  main  use  is  in 
the  validation  of  software  products.  A basic  approach  is  to 
first  decide  upon  the  desired  reliability  levels.  Next,  the 
ntmiber  of  tests  required  must  be  determined.  Actual  tests 
must  then  be  conducted  and  the  nximber  of  successful  tests  must 


be  recorded.  After  the  tests  are  completed,  the  decision 

; 

to  accent  or  reject  a nroduct  nniat  he  made.  An  e«aTRn1e  nf  \ 
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ACTUAL  COMPLETION  LEVEL 

ESTIMATED  VS.  ACTUAL  COMPLETION  LEVELS 

Figure  9 
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TYPICAL  GRAPHS  OF  REPORTED  COMPLETION 
STATUS  AS  A FUNCTION  OF  TIME 

Figure  10 
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the  curves  which  plot  pass  probability  versus  true  reliability 
are  shown  in  Figtire  11  for  a system  which  would  have  a relia- 
bility of  .85  and  an  acceptance  probability  of  .01.  Notice 
for  zero  failure,  28  tests  would  be  required,  one  fail\ire 
would  reqtiire  44  tests,  etc.  Extensive  test  data  on  actual 
project  has  been  gathered  to  validate  the  Acceptance  Model. 

STATUS 

DOMONIC  is  an  operational  system.  During  the  past  year, 
DOMCNIC  has  been  used  by  a number  of  staff  members  who  are 
developing  program  applications.  The  university  computer 
center,  the  DOMONIC  development,  and  the  university 
applications  development  are  all  activities  placed  under  the 
Data  Processing  Center.  The  D0M3NIC  system  has  been  developed 
by  one  area  and  used  by  a different  area  within  the  Data 
Processing  Center.  Over  35  docisnentation  units  (a  single 
project  is  defined  as  a docvonentation  unit)  have  \ised  the 
DOMONIC  system  in  the  development  of  five  new  systems  and  in 
the  maintenance  of  30  existing  systems.  The  systems  make 
up  2,950  programs  and  are  comprised  of  over  700,000  lines  of 
code. 

NEXT  PHASE 

As  mentioned  earlier,  the  DOMONIC  system  was  developed 
to  operate  in  both  batch  and  interactive  modes.  Vftiile  the 
interactive  version  operates  effectively  on  the  nonsattirated 


I 

I 

I 

Amdahl  470  V/6  at  Texas  A&M,  the  saturated  IBM  system  at  | 

Goddard  will  have  difficulty  responding  in  the  interactive  ■ 

! 

mode.  During  the  next  phase,  critical  program  modules  ! 

will  be  redesigned  to  speed  up  che  interactive  response  time.  I 

Another  approach  to  provdLng  response  time  will  be  to  distribute 
the  critical  interactive  operations  to  inexpensive  microcom- 
puters that  will  interface  to  the  operating  system  using 
standard  protocols  and  drive  \jp  to  16  terminals.  The  inqproved 
version  to  the  system  that  is  being  delivered  to  operate  on 
the  IBM  system  at  Goddard  will  be  adapted  to  operate  on  CDC 
and  Univac  computers  at  other  locations.  During  this  phase, 
additional  effort  will  also  be  expended  to  improve  and  refine 
the  software  reliability  models. 

There  are  a number  of  projects  at  the  Data  Processing  1 

Center  related  to  DOMONIC  development.  These  projects  are:  1 

1.  A COBOL  statistic  collector  is  being  developed  for 

1 

Rome  Air  Development  Center  which  can  do  static 
analysis  of  COBOL  programs  and  be  expanded  to  do 
dynamic  analysis. 

2.  The  building  and  testing  of  a system  to  randomly 
generate  test  data  to  validate  programs  has  been 
completed. 

3.  A comprehensive  analysis  of  program  connectivity 
to  determine  optimal  overlay  system  and  virtual 
memory  working  sets  has  been  completed. 
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4.  Comprehensive  study  of  program  complexity  has  been 


completed  which  includes  a rough  quantitative  measure 
of  program  con^lexity. 

5.  Design  specification  languages  are  being  investigated. 

6.  Line  routing  and  placement  routine  for  general  drawing 
systems  are  being  examined. 

DESIRABLE  ENHANCEMENTS 

The  generalized  flowchart  system  can  be  expanded  to  auto- 
matically produce  HIFO  charts.  Extensive  data  shoxild  be 
gathered  to  investigate  programmer  productivity.  A project 
evaluator  should  be  developed  to  automatically  determine  the 
status  of  a software  system  delivered  to  a contracting  agency. 
The  evaluator  system  could  predict  software  reliability  u.sing 
a model  such  as  the  Acceptance  Model.  Packages  that  automati- 
cally test  software  could  be  driven  by  the  system.  The 
delivered  system  could  automatically  analyze  a program  to 
determine  whether  structured  programming  techniques  were  used. 
Also,  the  system  could  be  automatically  checked  to  determine 
whether  appropriate  programming  practices  and  standards  were 
followed. 

There  has  long  been  a desire  in  both  hardware  and  software 
development  to  produce  a high  level  design  specification  lan- 
guage. Such  a language  could  be  developed  for  the  DOI^MIC 
system. 


576 


4ie.« 


ACICNOWLEDGEMENTS 


Numerovis  staff  members  and  graduate  students  have  parti- 
cipated in  the  design  and  development  of  the  DOMONIC  system. 
Evmlnious  Damon  at  NASA  Goddard  Space  Flight  Center  has  given 
major  encouragement  and  advice  throughout  the  DOICNIC  devel- 
opment. Louis  DeVito  and  Mike  Quick  deserve  special  recognition 
for'  their  contribution  to  the  DOMONIC  design  and  development. 
Susan  Arseven  supervised  the  initial  design  and  implementation 
of  the  first  version  of  the  system.  Pete  Marchbanks  has 
supervised  the  latest  design  and  development  of  the  current 
version  of  the  DOITONIC  system.  Jean  Zolnowski  and  Jimmy  Moore 
designed  and  developed  the  flowcharting  system  included  in 
the  DOMONIC  system.  Roger  Elliott  has  been  involved  in  most 
phases  of  the  system  design  and  has  supervised  the  development 
of  the  reliability  model.  Larry  Ringer  has  acted  as  consultant 
on  the  development  of  the  reliability  models.  Numerous  other 
graduate  students , faculty  members , and  computer  center  staff 
have  also  made  significant  contributions  to  this  development. 


DBS/ If /mb t 
08/17/77 


577 


DESIGNIX  A SOFTWARE  MEASUREMENT  EXPERIMENT* 


Victor  R.  Basili  and  Marvin  V.  Zelkowitz 
Department  of  Computer  Science 
University  of  Maryland 
College  Park,  Maryland  20742 

Overview  of  the  Laboratory 

The  Software  Engineering  Laboratory  has  been  established  at  NASA  Goddard 
Space  Flight  Center  In  cooperation  with  the  University  of  Maryland  In  order  to 
stiidy  the  effects  of  various  software  development  methodologies  on  software 
projects  in  a real  environment.  Projects  are  being  studied  at  three  levels  of 
control  and  detail:  screening  experiments,  send -control  experiments,  and  control 
experiments. 

The  screening  experiments  Involve  the  collection  of  data  on  the  development 
of  several  ground  support  systems  used  to  control  spacecraft  operations.  Their 
purpose  is  to  detendne  how  software  is  being  developed  now,  supply  data  for  a 
data  bank  of  information  to  classify  projects  for  future  reference  and  public 
availability,  expose  the  differences  between  the  theoretical  application  of  a 
methodology  and  its  practical  implementation,  discover  what  parameters  can  be 
validly  isolated,  expose  the  psurameters  that  appear  to  be  causing  major  problems, 
and  discover  the  appropriate  milestones  and  techniq,ues  that  show  success  under 
certain  conditions. 

The  semi -controlled  experiments  involve  the  collection  of  data  on  several 
similar  attitude  determination  systems  using  prespecified  development 
methodologies.  Their  purpose  is  to  extend  the  screening  experiments  by 

■m' 

permitting  tighter  control  and  to  permit  the  monitoring  of  different  development 
methodologies,  comparing  the  various  factors  affected  by  the  different 
methodologies. 


Research  supported  by  grant  NSG-5123  ftom  NASA/Goddard  Space  Flight  Center  to 
the  University  of  Maryland. 
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The  controlled  experiments  involve  the  development  of  duplicate  prograjos 
analysis  and  data  base  systems  with  fairly  rigid  control  of  many  of  the  factors 
affecting  software  development.  The  purpose  here  is  to  design  usable  experi- 
ments for  various  development  factor  evaluation  that  can  yield  statistically 
'/alid  results. 

The  activitif.s  of  the  Laboratory  include  data  collection,  data  processing, 
data  evaluation,  and  co\irses  on  methodology.  Data  collection  activities  include 
forms  filled  out  at  various  points  in  the  development,  covering  various  aspects 
of  the  product;  interviews  used  to  validate  the  forms  and  collect  information 
not  e2Lslly  filled  out  on  a form;  and  automatic  collection  of  data  via  the 
existing  system  and  program  development  library  when  available.  The  data 
processing 'activity  involves  entering  the  data  into  a computerized  data  base 
and  screening  for  accuracy.  The  data  evaluation  activity  involves  the  analysis 
of  the  data  collected  with  emphasis  on  management,  reliability,  and  complexity. 
This  analysis  deals  with  simple  raw  statistics,  in  the  forms  of  counts,  s\ims 
and  averages,  derived  statistics  in  the  form  of  correlations  and  multivariate 
analysis  of  the  relationships  between  various  factors,  and  the  development  of 
measures  based  on  theoretical  models  of  program  behavior  and  complexity.  The 
measures  will  be  computed  from  combinations  of  the  raw  statistics  and  derived 
statistics. 

The  simple  raw  statistics  include  project  attributes  such  as  type  of 
processing,  instruction  size,  code  size,  methodologies  and  technic^ues  used, 
resources,  total  man  hours,  calendar  time,  types  of  computer  access,  automated 
aids  and  tools,  usable  items  from  previous  projects,  milestones,  etc.  Also 
included  are  counts  of  total  number  of  program  changes,  total  errors  due' to 
various  factors,  such  as  misinterpretation  of  the  requirements  or  specification, 
total  time  spent  in  each  phase  of  development,  errors  found  using  various 


techniques,  total  computer  usage,  etc. 
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Derived  statistics  axe  used  to  find  relationships  among  various  factors. 

These  would  include  the  relationship  between  the  niimber  of  errors  and  program 
size,  between  errors  plotted  against  time  in  developement , between  various 
estimates  (e.g.,  cost,  time)  which  have  been  reevaluated  at  various  points  in 
the  development,  and  actuals  at  the  end  of  the  project,  noting  points  at  which 
the  estimates  become  realistic,  and  between  errors  distributed  according  to  the 
amount  of  time  they  remained  in  the  system,  etc. 

At  the  other  end  of  the  spectrum,  measures  are  being  developed  that  are 
associated  with  different  models  of  program  behavior  and  complexity  which  are 
being  tested  against  the  statistics  calculated  above.  For  example,  various 
measures  derived  from  models  of  complexity  will  be  compared  with  development 
time,  errors  and  subclassifications  of  errors,  subjective  views  of  the 

V 

complexity  of  various  modules,  etc.,  to  determine  if  these  measures  and  models 
can  be  validated  in  a real  environment.  Measure  of  complexity  include  the 
number  and  significance  of  control  paths  and  data  bindings  based  on  various 
criteria  for  hierarchical  decomposition  of  the  program  into  elementary  sub- 
3Ch«nes  using  different  formulas  to  combine  their  individual  evaluations  (SULL,73)» 
(me,  77). 

This  paper  deals  predominantly  with  the  purpose,  design  and  problems 
Involved  In  organizing  a controlled  experiment.  The  purpose  of  the  controlled 
experiment  is  to  Isolate  the  effect  of  specific  factors,  in  our  case  mostly 
software  development  techniques,  on  the  development  process.  The  idea  is  to 
design  as  airtight  as  possible  a valid,  usable  experimental  paradigm  for 
methodology  evaluation.  Clearly,  regardless  of  the  statistical  significance 
of  the  results  of  one  experiment,  one  needs  to  run  several  such  experiments  to 
accumulate  substantial  confidence  in  the  Interpolations  mads  ffom  those  results. 


Design  of  the  Control  Eyperlment 


Thb  current  design  involves  the  use  of  two  programining  teams,  both  assigned 
the  same  project  tasks.  Both  groups,  the  experimental  (l)  and  control  (2)  will 
be  treated  in  identical  fashion  as  much  as  possible,  with  the  sole  systematic 
exception  being  the  experimental  treatment.  The  experimental  group  will  be 
assigned  a task  A which  will  be  used  as  an  indicator  of  their  "normal"  behavior. 
They  will  then  be  trained  in  a specific  methodology.  The  methodology  will  be 
reinforced  in  a session  analyzing  the  development  task  A but  in  the  new  methodology. 
They  will  then  be  given  a second  task  B in  which  to  practice  the  methodology. 

They  will  then  be  assigned  task  C with  the  speci.fication  that  they  use  the  newly 
learned  and  practiced  techniques.  In  parallel,  a control  group,  group  2,  will 
also  be  given  task  A,  followed  by  session  analyzing  the  development  of  A using 
the  methodology  performed  on  A,  followed  by  task  B,  followed  by  task  C.  The 
difference  is  that  group  2 has  no  training  session. 

This  design  gives  us  several  points  of  comparison.  Ve  can  discover  any 
differences  in  the  capability  of  personnel  by  comparing  data  from  project  A for 
both  groups.  The  two  groups  can  then  be  more  honestly  compared  on  project  C by 
statistically  controlling  for  any  differences  observed  in  project  A.  The  design 
was  developed  to  minimize  several  problems  which  often  jeopardize  the  validity 
of  experimental  designs. (CAMP,  66). 

There  are  several  problems  which  affect  the  internal  validity  of  the 
experiments  i.e.,  extraneoxis  variables  which  can  produce  effects  confounded 
with  the  effect  of  the  experimental  manipulation.  There  is  also  the  problem  of 
external  '/alidity;  i.e.,  to  the  eT^eat  that  the  experiment  is  valid,  is  it 
possible  to  generalize  the  conclusion  drawn  from  the  experiment.  We  will 
discuss  several  of  these  problems  here  and  defend  the  design  of  our  control 
experiment  with  respect  to  these  factors  later. 
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training  will  be  discussed  first  and  then  techniq,ues  for  handling  specific  "real 
world"  problems  will  be  discussed. 
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The  experimental  group  and  control  group  will  each  consist  of  three 
programmers,  working  approximately  half  time  on  each  of  the  tasks  assigned. 

This  is  the  normal  operating  environment  at  Goddard;  most  programmers  work  on 
more  than  one  task  at  a time.  The  groups  began  task  A approximately  one  month 
apart.  Task  A consists  of  two  projects;  a FORTRAN  static  analyzer  and  a human 
resource  scheduler.  The  code  analyzer  will  generate  several  statistics  about 
FORTRAN  source  programs  and  is  estimated  to  be  a two  man  month  effort.  The 
human  resource  scheduler  is  an  interactive  data  base  system  to  enter,  update 
and  report  the  scheduling  of  people  on  tasks.  It  is  estimated  to  be  a three 
man  month  effort. 

After  both  subtasks  of  task  A have  been  completed,  the  experimental  group 
will  undergo  training  in  a particular  set  of  structured  design  and  programming 
techniques.  The  training  will  consist  predominantly  of  a one  week  course  which 
will  cover  the  topics  shown  in  the  following  outline i 

Day  It  Introduction,  functional  model  of  structured  programming, 

process  design  langu2ige,  reading  structured  code,  laboratory 
exercises  in  the  given  techniques. 

Day  2 1 Management  and  estimating  problems,  proving  correctness  of 
structured  design,  writing  structured  programs,  laboratory 
exercises. 

Day  3t  Documentation,  stepwise  refinement,  stepwise  reorganization, 

^ ' 

laboratory  exercises. 

Day  kt  Organizing  for  structured  programming,  modularization,  top 
down  development,  stubs,  program  development  libraries, 
walk  throughs,  laboratory  exercises. 


I 

I 

i 
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Day  5*  Project  control,  iterative  enhancement,  case  study  analysis, 
laboratory  exercises. 

This  course  is  offered  on  a continuing  basis  and  the  experimental  group 
will  be  members  of  a larger  class  of  approximately  25  people.  This  approach 
was  chosen  to  help  minimize  any  special  group  interaction  effect  on  the 
experimental  group. 

In  order  to  help  insure  that  the  group  understands  the  new  methodology, 
there  will  be  a one  or  two  day  special  training  session  with  the  group  in  which 
the  design  of  the  FORTRAN  code  analyzer  from  task  A will  be  reanalyzed  within 
the  framework  of  the  newly  learned  methodology.  To  compensate  for  this  special 
group  activity  and  design  analysis  instruction  the  control  group  will  be  exposed 
to  a similar  experience.  Even  though  they  will  not  be  given  a course,  they 
will  have  a similar  one  or  two  day  session  in  which  they  will  review  their 
design  of  the  TORTRAN  code  analyzer  with  respect  to  their  particular  design 
methodology. 

Clearly,  it  takes  time  and  practice  to  achieve  a reasonable  amount  of 
skill  in  the  use  of  a new  set  of  techniq[ues.  It  is  difficult  for  us  to  estimate 
how  long  it  will  take  for  the  programmers  in  the  experimental  group  to  acquire 
a sufficient  amount  of  skill  to  show  an  improvement,  if  there  is  going  to  be 
one,  over  the  previously  acquired  technique  skills,  before  we  can  attempt  to 
test  the  use  of  this  new  skill.  To  compensate  at  leant  partly  for  this 
uncertainty,  task  B will  be  assigned.  Its  main  purpose  is  to  give  the  experi- 
mental group  experience  with  the  newly  learned  methodology.  Task  B is  a 
NAMELIST  processor,  a prog:  ji  that  will  recognize  S/36O  FORTRAN  NAMELIST  state- 
ments and  convert  then  into  equivalent  non-NAMELIST  type  FORTRAN  l/O  statements. 
It  is  assumed  that  this  task  is  a one  man  month  of  effort.  Both  groups  will 
perform  task  B. 


I 
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Task  C will  'be  used  to  perform  the  compaxison  between  the  two  groups. 

It  also  consists  of  two  srabtasksi  a contractor  financial  report  program  and  a 


control  monitor.  The  contractor  financial  report  program  is  a data  base  I . 

analysis  system  similar  to  the  himan  resource  scheduling  program  of  task  A. 

The  control  program  is  a real  time,  interactive  attitude  support  subsystem  of 
six  man  months  effort.  This  provides  one  project  similar  to  one  developed 
without  the  new  methodology  as  well  as  a totally  new  type  of  project. 

A major  advantage  of  this  design  is  that  it  permits  us  to  analyze  the 
differences  between  the  two  groups  with  regard  to  task  A,  working  in  their  own 
self -defined  environment.  Measttres  on  task  C can  then  be  statistically  adjusted 
by  performance  on  task  A. 

During  the  entire  duration  of  the  control  experiment,  regular  meetings 
will  be  held  with  each  of  the  groups  to  answer  any  of  their  q,uestions  on  how 
the  reporting  forms  shoxxld  be  filled  out  and  to  gather  extra  Information  that 
is  either  not  clearly  specified  on  the  form  or  was  not  obtainable  from  the 
forms  at  all.  These  interviews  will  also  bo  used  to  help  validate  the  infor- 
mation that  is  on  the  filled  out  forms.  To  help  eliminate  any  biasing  here, 
ail  interview  questions  will  be  written  down  and  the  same  questions  will  be 
asked  of  both  groups. 

There  are  several  problems  that  arise  in  trying  to  implement  the  design 
in  a "real  world"  environment.  A couple  of  these  will  be  discussed.  One 
problem  that  arises  in  the  development  of  real  large  scale  projects  is  that 
the  specifications  are  not  always  well  defined.  In  this  case  a project  monitor, 

} 

who  is  not  a member  of  the  group,  will  be  used  to  answer  questions  about  the  ^ 

specifications  and  make  decisions  about  what  is  really  meant.  There  will  be 
two  different  project  monitors,  one  for  each  group.  This  is  to  help  guarantee 

that  any  information  learned  by  the  monitor  will  not  be  passed  on  unconsciously  f[ 

i i 

to  the  other  group.  When  a question  arises  about  interpretation,  raised  by  one 
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of  the  groups,  the  contract  monitor  will  consult  with  the  other  contract 
monitor  to  see  if  it  has  teen  resolved,  if  not  he  will  resolve  it.  If  it  has 
been  resolved,  he  will  be  told  how  it  was  resolved  and  then  give  a resolution 
to  his  group.  Thus,  if  the  methodology  used  by  one  group  helps  them  understand 
where  the  specifications  are  unclear,  earlier  in  the  development,  the  other 
group  will  not  benefit  from  this  information.  The  only  problem  here  is  that 
the  two  groups  may  develop  slightly  different  systems.  In  this  case  we  will  go 
to  outside  arbitrators  and  to  the  project  monitors  to  determine  that  both  1 

systems  meet  \iser  requirements  satisfactorily. 

There  is  also  the  problem  of  keeping  the  experiment  secret.  That  is,  we 
do  not  want  either  group  to  know  that  the  other  is  doing  the  exact  same  task  and 
that  they  are  being  compared  on  certain  measures  of  management  control, 
reliability  and  the  complexity  of  the  final  project.  For  this  reason,  we 
selected  programmers  from  Isolated,  but  hopefully  si  mi  groups.  One  group 
consists  of  programmers  employed  by  a contractor.  The  other  group  is  internal 
to  NASA.  They  have  both  been  told  that  they  au:e  being  carefully  monitored 
r because  we  are  interested  in  the  effect  of  the  forms  and  >rtiether  they  can  be 
filled  out  accurately.  They  know  another  group  is  also  doing  a "similar”  set 
I of  tasks  and  oiir  interest  is  in  whether  the  forms  get  filled  out  in  the  same  way. 

Defense  of  the  Design 

^ In  this  section  we  will  try  to  defend  the  design  and  implementation  of  the 

experiment  from  the  points  raised  by  Campbell  and  Stanley  that  threaten  the 
I internal  auad  external  validity  .of  the  experiment. 

The  effect  of  history  has  been  minimized  by  organizing  both  developments  in 
as  close  a time  period  as  possible.  Actually,  as  we  stated  above,  one  group 
did  start  about  a month  ahead  of  the  other  but  we  do  not  feel  this  will  make 
much  difference  in  a project  that  has  an  elapsed  calendar  time  of  9 to  12  months. 
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course  of  events  over  time  is  identical  for  toth  groups,  except  for  the 
experimental  group’s  training  session. 

Matinration  was  seen  to  have  the  potential  of  helng  a real  problem  in  this 
kind  of  research  since  the  more  one  knows  about  a particular  project  and  the  i, 

more  a group  works  together,  etc. , the  better  they  may  perform.  For  this  reason 
it  would  not  be  proper  to  compare  the  performance  of  a single  experimental  group  ^ 

a • 

on  tasks  A and  G,  even  if  they  were  similar  projects.  The  proposed  design, 
using  a control  group,  has  the  effect  that  the  maturation  factor  and  learning  i . 

curve  affects  both  groups  similarly.  The  experience  of  the  groups  is  kept 
similar,  especiailly  with  the  design  ainalysls  sessions  and  the  development  of 
task  B. 

With  regard  to  the  testing  factor,  there  is  no  explicit  testing  that  woxild 
Influence  either  group,  except  possibly  for  the  experience  of  filling  out  of 
the  forms  and  the  interviews.  However,  in  those  cases  both  groups  have  the  same 
experiences. 

The  variability  of  instrumentati on  la  kept  to  a minimum  by  using  the  same 
forms  for  both  groups  and  similar  Interview  questions. 

To  minimize  the  possible  effect  of  statistical  regression,  we  tried  to 

^ T 

choose  average  programmers  for  both  groups.  That  is,  we  did  not  choose  super 

i 

programmers  or  poor  ones.  This  rating  was  decided  based  on  general  knowledge 
of  the  programmers'  performance  on  previous  tasks. 

Because  of  the  need  for  some  secrecy  and  problems  with  contracting  for 
specific  people,  the  selection  variable  presented  more  of  a problem  than  the 
others.  Ve  could  not  make  a random  selection  of  programmers.  Vhat  we  did  do 
was  try  to  match  an  internal  group  to  the  contractor  group.  In  any  case, 
simple  effects  of  selection  biases  may  be  determined  by  examining  our  measures  ^ 

with  respect  to  project  A and  any  sv  :;n  effects  controlled  in  later  analyses. 


To  minimize  problems  with  exnerimental  mortality,  we  chose  projects  of 
minimal  size,  the  whole  development  lasting  somewhere  on  the  order  of  nine 
calendar  months.  We  tried  to  make  sure  the  participants  would  be  available  for 
the  full  dxiration  of  the  project,  bainring  any  unforeseen  disasters.  Clearly, 
there  is  no  way  to  totally  control  this  variable. 

Based  on  the  fact  that  the  two  groups  are  similar  in  terms  of  background 
and  capability,  it  is  hoped  that  the  select! on-matinrati on  interaction  will  not 
be  a problem;  that  it,  it  should  affect  both  groups  the  same  in  that  there 
should  be  similar  matiiration  for  both  groups  as  they  are  exposed  to  the  sequence 
of  tasks. 

Clearly,  the  reaction  or  interactive  effect  of  testing  is  a factor  here 
that  will  affect  generalizing  the  results.  It  is  not  normal  for  projects  to  be 
studied  to  this  degree  of  detail.  There  will  be  an  effect  related  to  the 
filling  out  of  the  forms  and  the  knowledge  of  the  participants  that  they  are 
being  studied,  even  though  they  do  not  know  the  evaluation  technique,  that  will 
not  be  able  to  be  factored  out.  For  this  purpose,  we  plan  to  collect  data  on 
several  projects  that  have  already  been  completed  but  were  not  monitored  so 
closely.  In  this  way,  it  will  be  possible  to  get  general  data  to  compare  with 
the  data  collected  on  the  control  experiments  and  see  if  they  are  performing 
within  the  general  set  of  standards. 

With  regard  to  interaction  effects  of  selection  biaises  and  the  exnerimental 


variable,  both  groups  appear  to  be  rather  typical  of  the  general  population  of 
programmers  and  there  appears  to  be  no  special  interest  on  the  part  of  either 
group  in  the  new  methodology. 

The  decision  to  analyze  the  development  methodologies  was  not  based  on  the 
belief  that  there  was  a specific  methodology  that  was  best  but  on  the  fact  that 
analysis  should  generate  a better  understanding  and  therefore  help  in  creating 
an  improvement  in  any  methodology.  Thus,  there  was  no  conscious  predisposition 
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toward  a partic^^lar  methodology  on  the  part  of  the  experimentors  that  would 
give  any  advantage  or  disadvantage  to  a particular  methodology  being  tested. 

In  fact,  the  specific  structured  prograonlng  methodology  being  tested  Is  only 
one  of  many  methodologies  aind  variations  that  are  to  be  tested.  It  Is  also  true 
that  the  programmers  undergoing  the  experiment  do  not  know  what  the  experl- 
mentors  are  looking  for  and  therefore  cannot  behave  In  any  particular  way  to 
affect  the  results  of  the  measures.  As  stated  earlier,  there  will  be  some 
reactive  effects  of  experimental  arrangements  based  on  the  programmers  knowledge 
that  their  environment,  due  to  the  forms,  is  different  than  the  normal 
environment.  However,  we  do  have  an  advantage  over  most  experiments  of  this 
type  In  that  we  are  trying  to  study  the  software  development  process  la  Its 
"natiiral''  environment  rather  than  In  a classroom,  student  programmer  situation. 

Lastly,  there  should  be  no  multiple-treatment  Interference  since  our  studies 
should  Involve  fresh  population  samples  of  programmers.  None  of  them  have  been 
used  In  earlier  experiments.  Hopefully  the  programming  community  Is  large 
enough  that  there  will  always  be  a fresh  supply  of  "uncontaminated"  subjects. 

Conclusion 

From  what  has  been  said.  It  Is  clear  that  It  Is  difficult  to  design  a 
controlled  experiment  in  a real  environment.  There  are  too  many  factors  that 
cannot  be  measured  accurately  or  completely  controlled,  such  as  programmer 
ability.  Designs  must  be  devised  that  adnlmlze  these  problems.  It  Is  difficult 
to  Implement  a design  since  It  Is  difficult  to  control  all  the  factors  In  a 
real  environment;  e.g.,  contracting  software  so  that  It  b«-performed  In  a 
certain  way,  guaranteeing  that  the  same  people  will  be  available  throughout  the 
experiment. 

However,  the  authors  believe  that  It  Is  Important  to  perform  such  studies, 
creating  the  moat  controlled  environment  possible.  If  we  are  ever  to  gain  amy 
real  insights  into  the  effects  of  various  software  methodologies  on  the 
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development  process  and  the  resulting  product.  We  feel  the  design  proposed  in 
this  paper  is  a reeisonable  first  attempt  at  approximating  an  effective  experi- 
mental design  in  the  given  environment.  It  is  clear  that  much  will  be  learned 
about  the  development  of  program  development  experiments  and  it  is  hoped  that  a 
great  deal  will  be  learned  about  the  development  process.  We  hope  to  continue 
I to  report  on  the  progress,  problems,  errors  and  successes  of  running  controlled 

experiments  in  this  area  and  hope  that  we  can  gain  from  the  efforts  of  others 
as  well. 


A cknowl edgement s 


The  authors  wo\ild  like  to  thank  Robert  Reiter  at  the  University  of  Maryland 


for  his  work  on  the  basic  controlled  experiment  design  currently  in  use. 


References 

(3ASI,  77)  Rasill,  Victor  R.,  ZelJcowltz,  Marvin  V,,  ^ , The  Software 

Engineering  Laboratory,  University  of  Maryland  Computer  Science 
Technical  Report,  TR-535,  May,  1977 » 10^  pages. 

(gamp,  66)  Campbell,  D.  T.,  Stanley,  J.  C. , Experimental  and  Quasi -Experimental 
Designs  for  Research,  Chicago,  Rand  McNally  Publishing  Co.,  1966. 

(LING,  77)  Linger,  R.  C.,  Mills,  H.  D.,  Structiored  Programming  Theory  and 
Practice,  Addison  Wesley,  1977  (to  be  published). 

(SULL,  73)  Sullivan,  J.  E.,  Measuring  the  Complexity  of  Computer  Software, 

MITRE  Corp.  Report  IfIR-2648,  Vol.  V,  June  1973. 

(ZELK,  77)  Zelkowltz,  M.  V.,  Basil! , Y.  R.,  Operational  Aspects  of  a 

Software  Measurement  Facility,  Proceedings  of  the  Software  Life 
Cycle  Management  Life-Cycle  Workshop,  August  1977. 


1 

S 


591 


Qpcrstlaoal  Aapscts  of  a Softvare  MsaaureBent  ?aelllt7* 


MurlJi  7 . Zalluwltz  and  Victor  R. 
lepartant  of  Co^pistar  Sclexice 
UnlTsrslty  of  Marylaxid 
CoUega  Burk,  MarylaxMi  2Crrk2 

Tba  OtaiTBrsit/  of  Murlaiid^  in  conjunction  with  RASA  Goddard  Spaoa  Tligtxt 
Cantar  has  orguilzad  tha  Software  Snglnaarlng  Laboratory  for  tba  purpose  of 
Wasaur-tng  and  evaluating  technlq.uea  used  in  the  darralc^Bast  of  spacecraft 
related  softmre  fBaalU-Zalkoirltz) . After  being  la  operation  for  a jear, 
we  are  now  starting  to  process  the  data  necessary  to  perf  on  these  evaluations 
ftom  shoot  a dozen  1I8A  projects*  Tlroai  the  experience  gained  by  collecting 
data  from  these  projects,  our  data  gathering  process  has  evolved  Into  a aore 
effective  operation* 

As  lASA  software  is  developed  auzilliary  data  for  the  laboratory  are 
produced  which  passes  through  four  distinct  phases:  project  developeant,  data 
coUsetian,  data  processing  and  data  evaluation.  This  report  describes  the 
flrgsnlzatian  of  the  laboratory  and  how  we  have  handled  sone  of  these  ppera- 
tional  aspects* 


j ; 

I 

I 


BAIA  yiow 

IISA/QBFC  lamches  apprcaclaately  2$  satellites  per  year  and  nost 

of  these  Include  the  developaeat  of  ground  support  software  requiring  about 

six  aan  years  of  effort.  These  software  projects  generally  take  about  a year 

and  involve  fren  8 to  1$  people,  nost  of  whan  are  working  on  several  such 

projects  stanltaneously*  The  projects  are  supervised  by  lASA/OSIC  personnel 

with  nost  of  the  actual  developnent  by  employees  of  an  outside  contractor. 

In  an  era  of  tl^it  budgets,  the  purpose  of  the  laboratory  is  to  Investigate 

these  projects  and  find  techniques  that  will  lead  to  Increased  productivity 

*-  Beseareh  supported  In  part  by  grant  SG>5123  froai  the  ■ational  Aeronautics  and 
Space  Admalstratioa  • Goddard  Space  Plight  Center  to  the  Qaiversl-^  of  Ihryland. 
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oa.  « eonstast  XMrly  'bodgst. 

wAflA  aal  contractor  aaMgaaBSt  were  eager  to  partlclpata  Is  this  studji 
hot  vara  cooBamed  that  the  on  currant  davalppannt  schedules  and  costs 

ha  mLBLml.  Ih  order  to  this  impact,  va  decided  early  to  use  a sat  of 

reporting  form  as  the  basic  data  gathering  asehaalsii.  It  vas  hoped  that  the 
foraa  could  be  filled  out  easily^  would  not  Interfere  with  current  techniques, 
and  would  not  Inrolre  nueh  overhead,  yet  would  still  glre  an  accurate  picture 
of  developnest  progress. 

The  laboratory  was  organized  Is  August,  197^  via  a research  grant  to  the 
IhilTerslty  of  Maryland.  Currently,  personnsl  include  3 eoployees  of  IASA/QBFC, 
2 faculty  and  9 students  at  the  Itelwerslty  and  approaclnately  73  progranaera 
eaplcyad  by  the  contractor  on  the  projects  that  we  are  aonltorlng.  Bie  two 
aajor  gf  ^ha  laboratory  are  project  "*•-  and  data  evaluation.  Tha 

day-to-day  operatloiial  aspeeta  currently  are  biroken  deem  Into  four  areea: 

1.  IA8A/<BIC  peraooael  are  the  Interface  between  the  contractor  and  tbe 
laboratory.  They  ace  responelble  for  project  developnent  sad  for  contractor 
coBQllaaee  with  lahoratory  requlreaents.  They  sre  supervising  spacecraft  pro- 
jects as  part  of  their  narasl  vorkload  In  their  aoraal  enTlronasat. 

2.  The  aaln  research  respoaslblUtles  are  being  perfomed  by  the  authors 

of  report  with  several  graduate  students.  Ih  addition  soae  of  the 

1A8A  persoanel  are  isfolved  is  this  activity. 

3.  Ih  order  to  evaluate  the  collected  data,  laboratoary  software  for  a 
yom-based  systea  Is  being  developed.  This  Is  being  perfoznsd  by  two  student 
progriMcrs  under  faculty  superrlslon. 

k.  day-to-day  data  eoUsctlon  Is  currently  being  handled  by  one  USA 
employee  and  four  unlwarslty  undergraduate  atadenta  under  the  si^ervlslaa  of 
a graduate  student. 

(Uvea  this  struetnre,  data  flows  through  ths  laboratory  as  ontUnsd  by 
figure  1: 


— — I 

5«  Qm  qymriM  are  run  against  our  collected  data  and  etatlatlea  about 
project  attributes  are  produced  (1.7).  Sbe  output  la  eltber  a Ust  of  factors 
folflUlnc  tbe  qperj,  a table  at  statistics^  or  a plot  of  one  fbetor  tstsus 
anotber  (fenermlly  soae  factor  Tersus  tine) . Soae  of  tbe  factors  that  ve  are 
aeasurlng  include  size,  cost  and  tins.  Less  *»*ff*'**''*  factors  Include  Isrel 
of  specifleatlan  and  dereloparat  technique  used. 

Xatar  sections  of  this  report  will  describe  aost  of  these  steps  la  greater 
detail. 

Host  of  tbe  fan  of  1976  eas  derated  to  dereloplng  tbe  set  of  reporting 
focne  to  capture  tbs  data  on  each  project,  with  tbe  first  projects  subnittlng 
ccBBlated  foms  as  of  Peoeaber,  1976.  k set  of  seren  foms  bare  been  dereloped. 

i These  fores  eon  be  dlrlded  Into  tvo  classes-  tbcee  that  describe  attrlbutea 

I 

[ of  a project  and  are  filled  out  relatlreljr  Infrequentljr,  and  those  that 

aonltar  progress  and  are  flTLed  out  frequsntlj.  The  fores  that  describe  at- 


tributes of  a project  are  tbe  fallowingt 

1.  Qsiaral  ftoject  Sueearr.  This  fore  la  filled  out  at  each  project  alls- 

stone,  and  there  are  iron  fire  to  tea  eilsstonss  in  the  llfatlns  of  a project 
(srerT’  six  to  ei^^  weeJni).  This  foxs  describes  tbe  orerall  project  structure, 
and  the  techniques  xised  la  Its  derelqpnsnt.  Such  factors  aa  project  size, 
co^lerlty,  estlnated  ailestoaes,  requlrad  standards  and  docuaentatlon  are 
glrea.  Bon  these  over  tine  Is  one  of  the  areee  that  «e  are  Investigating. 

2.  Conponset  8u— rr.  This  fora  la  slxUar  to  the  gsaeral  project  suHsary 

except  that  It  describes  only  ana  saall  section  at  a systea  (e.  g.,  function, 
s^dgoutine,  COMIS  block,  etc.).  This  fom  Is  filled  out  vhea  tbet  pertlenlar 
eavcaeat  Is  first  daslgisd  la  ths  dsslgs  phase  of  the  project  and  again  ^ 

ehan  It  Is  eoeplstsd. 

4 d 

3*  PirngraMwr  iaalrst  aurrey.  This  fora  la  used  to  eollsct  gsaeral  back-  _ 

t 

ground  iafornatiaa  on  project  parscanel,  end  la  filled  out  only  once  by  seek 
prngraanar.  It  is  esed  to  eonpete  e profile  of  project  persasral  In  order  to  p 
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Tm  abla  to  eoi^are  tvo  projeeta  aora  equitably. 

Ibe  fozaa  that  aoBltor  project  progreaa  are  tbe  foUovliig  four: 
k.  Baaouree  3u— ry.  Tbla  fora  la  filled  oat  weekly  by  the  project  aaiiacer, 
oad  It  Uata  tbe  taoura  apest  by  each  ladlwldual  oa  tbe  project.  Since  tbla 
iafarmtloa  la  already  collected  for  aecountliig  purpoaea,  little  additional 
effort  was  added,  fix  addition,  lASA/SSFC  adda  to  tbla  fom  a eouat  of  tbe 
total  counter  tine  uaed  by  tbe  project  and  tbe  total  ma^bmr  of  runa,  aa 
reported  by  tbe  lASA  eoevotatlon  center. 

5.  Ccnpruent  Statue  Heport.  Tbla  fom  la  filled  out  weekly  by  each  person 

on  the  project.  It  Uata  the  miiOer  of  boors  spent  on  each  conponant  and  the 
aetlTlty  perfomad  on  that  eoe^onant  (e.  g.,  code,  test,  etc.).  Besides 

sons  of  tbe  obwloos  statlatlca  we  expect  to  obtain  (e.  g.,  nubber  of  boors 
spent  In  design  ws.  teat  ts.  code,  etc.)  ve  can  also  use  this  fom  to  werlfy 
tbe  teebnlqbes  uaed  In  project  dewelnisnnt.  Unfortunately  tema  Ilka  ”tqp  down," 
"structured  programing,"  "walk  tkrougbs,"  and  "code  reading”  tanwe  beeane 
ganaral  purpose  bnszwords  with  different  Interpretatloae.  Otae  of  tbe  reeulte 
that  wn  expect  to  get  fTon.  tbe  conponent  statue  report  la  to  be  able  to  elaaslfy 
the  tecbnlqoee  that  are  being  ueed  rather  than  tbe  ones  we  are  told  are  being 
used . We  can  obtain  tbla  laf omatlon  by  noting  wbaa  certain  aetlTlties  occur 
for  each  eonponent. 

6.  CospBter  Ptcgren  Bun  dnalyela.  This  fom  la  flUad  out  for  each  co^uter 
run.  Zb  la  aostly  a ebsckllat  of  aetlTltlaa  perfomad  (e.  g.,  eo^lla,  exeeuta, 
utility,  ete.).  Si  addition,  the  error  nassega  and  conponant  tendnatlag  tbe 
raa  la  Hated.  This  will  be  correlated  with  tbe  foUowlng  ***«*"gr  report  fom. 

7>  Cbaane  Beport  Fom.  ?or  each  la  tbe  source  code  for  a eavoant 

(either  to  fix  ea  error  or  to  l^pleaant  sona  ebaaga  la  apaelflentlons  or  daslgn) 
a rhaage  report  fom  la  scbnlttad  listing  tbs  ebangt  and  tbe  reason  for  It. 

Since  a fom  of  tbla  type  has  been  la  use  for  sons  tine  in  this  particular 
MSb  eawlroaneat,  Uttla  additional  work  was  addad  to  existing  USA  projects. 

Oba  UadLtntioa,  kowewer,  la  that  are  not  reported  for  any  eo^oaent 
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until  that  eonpoMat  1m  placed  Into  the  project  library.  Tbna  vhlle  ve  are 
getting  all  of  the  errors  found  la  lategratloa  testing,  ve  are  not  (except 
for  a few  of  tbs  ncaltored  projects)  getting  nueb  firoa  tbs  nodule  design  and 
deTalopnent  phases. 


BgnL  ffnrJnnjKai 

Broadly  stated  tbn  purpose  of^  the  data  eoUaetlaa  pbase  Is  to  traasalt 
the  raw  data  on  each  nonitored  project  to  the  laboratory  for  processing  and 

analysis.  This  aspect  necessitates  the  Interaetloa  of  three  different  groups 

/ 

of  IndlTlduals.  Bsedless  to  say,  altbough  la  the  long  run  all  these  groups 
are  Interested  la  better  software  aathodologles,  each  bare  differing  lansdiate 
goals. 

The  prlaary  reqnlrenent  for  each  nonitored  project  la  to  derelop  spacecraft 
support  prograas;  therefore,  the  prlaary  goal  of  the  HBA/OSTC  persooneL  la  the 
Bsaagensat  of  project  derelopneat  on  schedule.  Boverer,  actual  progran 
derelopnsnt  la  by  aa  outside  eoBtraetcr  whose  nala  goal  la  to  dellrer  such 
projecta  on  tine  la  order  to  nset  contractual  dbllgatloos.  Finally  the  third 
group  larolrsd  la  the  laboratory  Is  the  IMrerslty  of  Murylaad  which  la 
prlaarlly  Interested  la  the  analysis  and  neesureaent  of  the  derelopnsnt  process. 

The  use  of  three  different  groups  has  both  negatlre  and  poslttre  attributes 
towards  our  goals.  Clearly  there  are  eoswaaleatloas  prbbleas.  iay  changes  la 
raportlag  reqplrensnta  aust  first  be  eonreyed  to  lASd/OBtC  who  la  turn  nust 
lafora  the  eontraetor.  Slallarly  any  uafarastlon.  tram  the  contractor  la  first 
routed  through  IdSA. 

On  the  other  hand,  this  situation  laereases  the  ii^artlallty  of  the  research 
groqp  with  the  eontraetor  group.  Since  we  are  Independent  of  lASh,  we  are  not 
lawolrsd  la  ratlag  eontraetor  perfomsaee  sad  ewaluatlag  contractual  arrange'^ 
neats.  Ve  seen  to  have  the  contractor's  trust  which  we  bellewe  glTss  us  nore 
accurate  date  to  work  with.  As  part  of  this  lapartlallty,  we  are  earefnl  not 
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to  roport  back  aaj  data  that  eaa  he  Identified  to  epeelflc  ladlrlduale.  All 
pgrograMBT  aaaes  are  eaeoded  la  onr  data  haee,  aad  all  data  reported  back  to 
the  eoBtraetor  (aad  to  lASA  also)  la  of  a saanarj  oattEre. 


pm  aocaasiaa 

Si  the  data  preeesslxig  phase,  the  data  on  the  projects  aast  be  traaserlbed 
oato  eoAlav  sheets  aad  estered  lato  a EPEU  data  base.  Sock  Issoes  as  data 
fallditj  aad  eo^leteaesa  are  of  prlaary  laportaaee  In  this  phase. 

The  raw  data  Is  cheeked  four  tines  before  being  Mtared  Into  the  data  base. 
The  fauns  are  first  encoded  oato  coding  sheets  aad  then  cheeked  for  errors 
by  a second  person.  The  eodlag  sheets  are  then  tjped  Into  the  eoqxiter  aad  ron 
throng  a special  prograa  ehieh  ftxrthar  cheeks  for  errors  (e.  g. , fomat  errors 
or  liproper  fields  such  as  a srong  component  naae  In  a certain  project)  and 
eoBTerts  the  data  lato  a forvat  snltable  for  Inelnsloa  lato  the  data  base. 

The  output  of  this  prograa  is  again  Tsrlfled  against  the  coding  sheets  to 
farther  check  for  tjplng  errors.  By  the  tlae  that  the  data  Is  la  onr  data  base 
ve  are  eoafldent  that  shot  hes  been  entered  Is  an  aceorate  traaslatlos  fron 
the  original  foms.  Vhether  these  foms  aecnrate  describe  the  aetaal  project, 
hoeerer,  Is  another  Issue  which  will  be  discussed  shortly. 

JotwB  validity  Is  perhaps  the  hardest  prdhlen  that  nest  be  tackled.  Qtafart-> 
nnately  there  ore  few  general  standards  applylag  to  all  IM3A/QBIC  projects, 
aad  one  of  the  re^tlrenents  on  the  Ishoratory  Is  to  naaenrs  current  techaiqnes 
vithont  Inpaetlng  nost  projects.  Thus  foms  are  filled  oat  vl^  warylag  degrees 


of  eonpleteasss,  aad  we  nest  be  carefal  la  applying  results  scross  seraral 
projects.  Pa  one  such  case,  there  Is  shout  a factor  of  three  la  aa^er  of  foms 
siteitted  between  one  project  end  othera  of  n slnllar  else. 

The  Inpeet  on  projects  brings  to  nlad  tvo  exaqleo  of  tha  kinds  of  prbhlsns 
we  are  feaed  wltk.  OSe  project  nanager  has  been  relnetant  to  pnrtielpnte  since 
hls  project  In  late  aad  ka  seas  no  apparant  baaaflt  iron  participating  in  this 
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rMiMTcii*  VhlLa  this  'blBMS  tbm  fora  tor  uy  locreMcd  latMwss^  tbty 

v«re  ■TrmilT  selMdalft  iMfocv  tbs  fom  were  laa^ltut^d.  XISA/GSFC  l>«lleTM 

thttt  tbiii  group  has  alvajs  Tmob  poorlj’  orguilzod  aad  tbs  probloM  vltb  ttaa  foroa 
only  poljxta  out  tbla  problea  aad  la  nort  tba  eauae  of  It.  7or  vail  aanaged 
projaeta,  tba  flUlag  out  of  tha  faroa  la  no  aora  eo^plez  than  aay  otbar 
adadalatratlTa  datall  of  a taak. 

Th-t*  sjiovs  cam  of  ttaa  dangara  la  lactarpratlag  our  eolL»«tod  data. 

Slaca  foraa  aay  ba  tba  flrat  to  go  irtiea  a projaet  faUa  bahlad  aebadule,  or 
alaea  tba  aora  orgaalxad  prograaaliig  alad  vUl  proeaaa  foraa  aora  aeetontaly, 
tba  data  aay  ba  blaaad  toaarda  projaeta  tbat  are  on  tlaa.  Zhla  aay  laad  to  tha 
laaitetastlatad  poaitloa  that  alaply  flUlag  oat  of  foraa  balpa  projaeta  raaala 
OB  aebadola.  VbUa  va*  ballaaa  that  this  oay  ba  tna.  It  vill  ba  difficult  to 
elala  aoch  Istarpratatloos. 

2a  aaothsr  azparlaaot  too  daraloiEBBits  of  a task  vara  oartartahan 

la  ordar  to  coapara  aaa  groiqp  aot  ualag  aay  spaelal  aathodology  vlth.  a aaeood 
group  ualag  a praspaelflad  tachalqpa  (Baalli^alkovlts).  Oha  of  tha  groepa, 

-tm  that  thay  vara  balag  lavaatlgatod  but  aot  azaetly  sura  how,  vaa  ertxaaaly 
rnnramad  about  parfaxaaaea.  Ihay  daeldad  to  ehaaga  thatr  aoraal  baharlor  by 
ualag  aspUeit  raqulxuBaata,  good  projaet  eoatrol,  top  dooa  daralopBoat,  etc. 
Boaavar,  «a  vaatad  to  aoBltor  aoraal  prograsa  for  this  group,  aad  thla  ehaaga 
vaa  aot  thslr  aoraaL  aartronaat.  Tortiaataly,  tha  usa  of  thaaa  "bow  Idaaa" 
laotad  about  a vaak  aad  tha  groip  raturaad  to  Its  ataadard  taehaiquaa  aad 
aatabllahad  aathodology  of  prograHdag.  Slaca  they  vara  still  aad 

darlfyiag  tha  raqpiraaaBta  for  tha  task  aad  had  aot  bagua  aay  daalga*  tha 
lipaet  OB  tha  aatlra  projaet  should  ba  alalaal. 

Bm  lagoet  of  tha  foraa  thaasalTsa  la  soBOthlag  idkleh  va  vould  Uks  to 
but  eammat  Igaora.  lha  cootraetor  has  asksd  for  a 10)(  laeraasa  la  eosts  Just 
for  flUlag  out  of  this  data.  Vhlla  va  do  aot  ballanra  that  suck  aatlaatas  ara 
raallatlo,  IMBd  has  atraad  to  It  for  aov  la  order  to  dovalop  a valid  data  base. 
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Aaotbrnr  problaa  vltli  tb»  foras  tb«lr  gsaaraUtx.  Da  order  to  beve  e 
•taaderd  data  bate,  a geaeral  porpoee  set  of  foxv  bai  been  developed.  Qifart-' 
nnately  aot  every  qneatlon  £a  appUeable  la  all  sltuatloos.  Thin  adda  arverbBad 
to  filling  out  the  fane  elace  prograners  do  not  knov  bov  to  aaever  soae  of 
the  qaestlane.  la  oar  next  Iteration  of  the  fane,  ve  Intend  to  ebatract  the 
-^rpleal  aaavera  being  given  at  preaeatt  and  nake  the  foxaa  non  of  a ebeekllat. 

(Uven  tbe  eoiqlated  fona,  tba.nesct  atep  vaa  to  encode  tbaa  Into  a BCU 
data  beae.  Ve  ebooe  to  tiae  tbe  aOBP  relational  data  baae  apatea  (Bald)  tbat 
rnna  under  tbe  LLBXI  operating  ajaten.  Zhia  deelalon  voa  nootly  for  Ita  coat 
itT3)  compared  to  other  relational  data  baae  ayrtean  ($100;000) ; however.  It 
did  hove  a good  reputation  f^oa  other  naera.  Itvortant  In  any  data  baae  project 
la  tbe  need  to  gat  data  Into  and  oat  of  the  ayatea;  boaever,  vltb  tbe  large 
aaoant  of  data  being  coUseted,  tbe  tendency  la  to  Ignore  output  la  an  atteagt  | 

to  keep  tbe  lapot  gntes  clear.  Zbla  la  clearly  a danger,  and  we  believe  (ao 
far  aaeceaafally)  tbat  nalng  a relational  data  baae  ajatea  allowa  for  complex 
output  querlea  to  be  eaally  written  to  aolve  tbla  prbblea. 

2h  oer  Inveatlgatlona  of  data  baae  ayateaa,  IBGBB3  waa  anppoeed  to  be 
relatively  bard  to  Inatall  bat  waa  aiQpoaed  to  work  well.  After  aoae  Initial 
tqe  probleaa  and  local  hardware  aalfanetlona^  ve  found  tbla  to  be  true.  Ve 
bad  little  probleai  inatalUng  tbe  ayatea;  bowever,  la  trying  to  adapt  it  to  ^ 

oar  nae  ve  did  dlaeover  aeverel  Ualtatlona » TlQin  la  aoaawbat  of  a toy 
ayatea  end  data  relatione  aaot  be  kept  aoaavbat  alB[pla  ($00  bytee  and  k9  flelda  I | 

per  entry) . la  addition,  the  ayatea  oeea  aany  of  tbe  feataree  of  TUZZ  idileb 
aakae  It  alow  oa  oar  6kk  ISFU^^.  Bila  bee  partially  rootrlctod  our  activity  | j 

bat  bee  not  forced  ue  to  our  overall  goala  or  proeednrea. 

Once  la  tbe  data  baae,  tbe  flrat  taek  baa  been  to  preaerve  data  validity.  J ] 

■ ! 

Ik  addition,  apeclal  prograaa  have  been  written  to  obeck  for  -twg  data. 

I 

Tor  exaivle,  alnee  foraa  are  oalqaaly  naidtered,  alaeing  nurtara  ore  eaally  | 

apotted  and  oaaally  aaaa  fooraa.  Aiao  aoaa  foma  are  to  be  filled  out 
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v«ttkljr  *0  it  ia  maaj  to  spot  ailsslBg  ones  in  that  saqusncs. 


HMSk  miflATiai 

Bio  data  oraluatlon  pbase  eoaulsts  of  doroloplBg  asasores  to  oraltiato  tbe 
■athodologieo  usod  oa  the  projects.  One  teehnlqpe  reconsended  hy  tbs  contractor 
aaa  to  prorlde  Instant  feedback  of  the  data  on  the  fame.  Bils  sas  to  hal^ 
keep  op  the  enthnaiasB  of  the  prograrntrs  for  the  entire  endearor.  Since  ve 
did  not  eaat  to  report  hack  any  Inf omatloa  that  coold  he  oaad  for  personnel 
enU.xiatlon,  ve  designed  sons  general  pnrpose  reports  vhleh  give  avmmrj  totals 
about  each  project. 

The  next  step  was  to  start  developing  ncore  eosplex  validity  checks  on 
the  conputerized  data  hose.  This  has  bean  the  najor  e^hasis  of  the  Isborstory 
for  the  last  few  aonths.  Dos  to  the  large  variation  ia  progranssr  perfamanee, 
we  are  trying  as  noeh  as  possible  to  sake  sure  that  what  ve  have  is  corToet. 
Cohering  slnllar  data  in  different  ways  on  diff ereat  foms  is  one  way  ve 
are  checking  for  eonsisteacy.  For  prograaner  tinas  tsom  the  resource 

nuanary  and  fron  the  coagonent  status  report  can  he  eoapared.  Also,  error  ms 
from  the  co^uter  progran  run  analysis  fom  can  he  coiqpared  to  the  auaber  of 
change  report  fozns  subaltted.  Looking  at  gross  data  on  projects  that  have  not 
haen  aonitorad  is  another  teehniq:oe.  Also,  intervlevs  with  sane  of  tbe  eon- 
tractor  psrsonasl  has  heea  used  to  validate  certain  answers. 

TTOC  queries  are  aov  halng  written  to  extract  detailed  Infomation  fkoai 
the  encoded  data.  A plotting  has  heea  lapleneated  to  plot  project 

characteristics.  «e  are  currently  at  the  stage  vtaare  sons  of  these  queries 
are  being  tested  on  the  data  base. 


WTWTTr 

As  this  report  describes,  the  Software  fttglnsarlng  Laboratory  has  haen  la 
eperatioa  for  a year  and  is  elsarly  a very  eosglex  operatioa  reqniring  constant 
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«tt«Btian  to  datoll  and  solylnc  the  waaj  sBall  and  large  practical  prcibleia 
aa  they  arlae.  Moore  effort  than  ve  orlgiaally  planiBd  la  going  lato  day-to- 


! 

i 


day  apara^ioa  aid  aaiotesaace.  Jarm  and  data  haae  Talidity  have  hecoae  the 
critical  aeetioa  throo^  vhieh  all  labaratory  activities  nov  depeed. 

Me  have  been  develotplag  a prototype  laboratoary  vhieh  caa  he  used  to  perfora 
the  needed  expert aentatiott  ia  order  to  develop  theories  of  program  develapmeat. 
dithnetfi  devalopad  vtthla  the  IdflA  framework  for  software  ilsiiil  nimsiil , we  do 
act  believe  that  the  prohleas  we  eneoaatered  or  oar  solutioas  to  them  are 
OBiqpe.  Qivea  seek  care  ia  aalirtalnlng  the  data,  we  believe  that  the  data 
which  is  being  eoHscted  is  valid,  usefol,  and  should  lead  to  laqportaat 
laslglits  about  program  development. 

eyeuem  J8 

CBasili  et.  al.)  Basil!  T.,  Zelkowitz  M.,  et.  al.,  Ihs  Software  Xaglmeerlag 
labaratory,  Seehaical  Bepoort  TB>535>  Coapoter  Science,  Oaieersity  of  Nuyland, 
Ifcar,  ISTT- 


(Basili-Selkowitx)  Bosili  T.,  Selkovitz  M. , Besi0iing  a software  measurement 
experlaeat.  Software  Life  Cyele  Management  Vorkshop,  August,  1977* 

(Mwdd)  laid  Stonsteaker  M.,  Vomg  1.,  a relationml  data  base  system, 

fttloeal  Caagater  Ccnfarence,  IST;,  pp.  409-416. 
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SOFTWARE  PSYCHOLOGY: 
SHRINKING  LIFE-CYLE  COSTS 


T.  LOVE 


General  Electric  Co 
Informations  Systems  Programs 
1755  Jefferson  Davis  Highway 
Arlington,  Virginia  22202 


Early  predictions  that  "thinking  machines"  would  increase  the  unemployment 
rolls  are  being  shown  to  be  incorrect.  In  fact,  a recent  forecast  suggests  that 
"by  1985,  automated  information  processing  could  be  the  most  labor-intensive  of 
all  industries  with  the  possible  exception  of  agriculture".  Who  would  have  ever 
predicted  that  we  may  someday  end  up  with  more  programmers  than  farmers? 

In  the  military,  government,  and  industry,  the  high  cost  of  software  is 
strictly  a function  of  the  labor  involved  in  producing,  operating,  and  maintaining 
code.  Then  once  we  have  produced  software  by  this  labor-intensive  and  expensive 
process,  frequently  it  does  not  do  what  it  is  supposed  to  do.  Little  wonder  that 
when  the  "All  Purpose  Elixir"  (structured  programming)  arrived  on  the  market, 
lines  formed  in  every  direction  to  partake.  However,  as  with  most  miracle  cures, 
this  one  has  not  lived  up  to  its  initial  advertising  claims. 

What  the  structured  programming  uproar  may  have  done  is  to  define  the  problem 
for  the  first  time.  That  problem  is  that  software  problems  are  people-problems 
which  cannot  be  solved  by  the  simple  addition,  of  a faster  black  box  on  the  computer 
system.  This  paper  will  describe  an  approach  to  solving  these  problems,  and  describe 
some  results  from  its  application. 
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THE  CASE  FOR  SCIENCE 


Engineering  methods  demand  some  underlying  set  of  rules  and  facts  which  can 
be  followed  in  a disciplined  way  to  produce  the  desired  result.  Discipline  alone 
is  really  not  sufficient.  A group  of  engineers  designing  a bridge  need  to  have  the 
same  theory  of  the  effect  of  gravity  and  of  the  stress  that  the  chosen  materials 
will  tolerate.  Newtonian  physics  and  carefully  controlled  stress  experiments 
provide  the  basic  rules  which  engineers  use  to  design  bridges  so  that  they  work 
the  first  time. 

In  software  engineering,  we  do  not  know  the  underlying  rules  and  facts  that 
we  need  to  do  our  work.  Instead,  we  are  busily  imposing  discipline  and  structure 
to  our  software  production  processes  without  any  understanding  why  a particular 
discipline  or  structure  should  be  preferable  to  another.  Furthermore,  the  number 
of  possible  disciplines  and  structures  is  sufficiently  large  that  evolution  to  better 
methods  through  trial  and  error  may  take  centuries  (assuming  that  we  maintain 
careful  records,  which  of  course  we  do  not)! 

The  ultimate  success  of  software  engineering  will  depend  upon  the  development 
of  the  underlying  scientific  knowledge  to  answer  our  present-day  "why"  questions. 

Due  to  the  inherent  complexities  built  into  .these  questions,  it  is  clear  that  no 
one  discipline  alone  can  provide  the  answers.  Some  disciplines  which  antedate 
computers  can  be  of  significant  benefit  by  at  least  providing  some  established 
methods  for  answering  our  questions, if  not  some  relevant  theories  to  guide  our 
question  asking.  Two  such  disciplines  which  have  been  used  in  the  research  reported 
in  this  paper  are  psychology  and  statistics. 

Therefore,  improving  software  production  and  maintenance  can  be  done  in  a 
rational  systematic  fashion  using  knowledge  and  technology  which  is  well  established. 
In  this  way  software  life-cycle  costs  can  be  reduced  without  reliance  upon  either 
our  fallible  intuitions  or  evolution.  Some  recent  examples  of  this  approach 
will  be  described  below. 

He  are  currently  conducting  experiments  for  the  Office  of  Naval  Research 
to  examine  four  phases  of  the  software  life-cycle:  (1)  understanding,  (2)  mod- 
ification, (3)  debugging,  and  (4)  construction.  A preliminary  experiment  has 
been  conducted  to  test  various  influences  on  the  human  understanding  of  software 
(Sheppard  and  Love,  1977). 

In  this  experiment  eight  experienced  programmers  were  asked  to  study  FORTRAN 
programs  and  reproduce  functionally  equivalent  programs  from  memory.  Understanding 
was  defined  as  the  percent  of  functionally  equivalent  statements  correctly  recalled. 
The  programs  were  wHtten  using  three  levels  of  control  flow  complexity  and  three 
levels  of  meaningfulr.ess  of  variable  names.  The  most  meaningful  variables  had  four 
to  six  characters,  and  the  least  mnemonic  ones  consisted  of  randomly  chosen,  single 
letters.  All  programs  were  written  in  standard  FORTRAN,  without  any  of  the 
popular  structured  preprocessors.  Meaningful  ness  of  variable  names  was  manipulated 
Independently  of  control  flow  levels.  The  experimental  design  allowed  each 
programmer  to  study  and  recall  one  version  of  each  of  three  programs  and  all  levels 
of  the  two  primary  independent  variables. 

An  analysis  of  variance  showed  no  significant  effects  due  to  mnemonic  levels 
(see  Figure  1).  Curiously  enough,  the  trend  was  toward  improved  performance 
with  random  letters  ~ exactly  the  opposite  of  our  prior  predictions!  However, 
significant  differences  between  control  flow  levels  were  found  with  the  more  struc- 
tured programs  being  the  easiest  to  recall. 
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MNEMONIC  LEVELS  CONTROL  FLOW  LEVELS 


Performance  was  also  compared  with  Halstead's  E,  a measure  of  the  effort 
required  to  write  a program.  A high  correlation  (-.31)  was  found.  This  experi- 
ment represents  the  first  time  that  Halstead's  theory  of  Software  Science  (1977) 
has  been  rigorously  tested  with  proper  experimental  controls.  Considering  that 
we  can  account  for  "nore  than  60%  of  the  variance  while  ignoring  differences  in 
use  of  mnemonics  ana  function  of  the  program,  suggests  that  our  intuitions  about 
programmer  behavior  may  not  be  so  accurate.  In  order  to  verify  and  gain  greater 
confidence  in  our  initial  findings,  a more  extensive  study  involving  36  subjects 
and  81  programs  is  iii  progress.  In  this  study,  great  care  is  being  taken  to  use 
programs  representative  of  those  encountered  by  professional  programmers.  Also 
the  experimental  design  will  precisely  determine  all  primary  effects  and  inter- 
actions which  are  of  interest. 

In  sunmary,  GE  is  demonstrating  that  experimental  studies  of  programmer 
behavior  can  be  done  and  the  nonintuitive  findings  of  our  research  suggest  that 
more  experimental  work  should  be  performed.  GE's  work  for  ONR  is  unusual  in 
the  software  research  field  in  that  it  is  testing  precise  predictions  of  a theory 
using  controlled  experiments. 


INDIVIDUAL  DIFFERENCES  IN  PROGRAMMER  PRODUCTIVITY 

Empirically,  we  have  begun  to  accept  the  notion  that  two  programmers 
with  equal  experience  and  credentials  may  program  at  radically  different  rates 
(up  to  25  to  1).  What  we  do  not  yet  know  is  the  distribution  of  these  differences 
and  why  they  are  large.  A field  study  will  be  described  which  addresses  these 
questions. 

In  an  academic  introductory  programming  class,  four  generic  types  of  in- 
formation were  obtained  to  answer  these  questions.  This  information  included: 

1.  Classroom  performance  measures  (grades,  test  scores,  etc.) 

2.  Unobtrusive  measures  from  actual  programs  (number  of  runs,  number  of 
changes  per  run,  type  of  change  made  to  program,  type  of  instruction 
changed,  etc.) 

3.  Questionnaires  from  each  run  (environment,  expected  time  of  completion 
of  problem,  source  of  assistance,  type  of  error  encountered,  confidence 
that  program  will  run,  etc.) 

4.  Measures  of  human  information  processing  (memory  span,  memory  for  programs, 
speed/accuracy  of  comparing  digit  strings,  etc.) 

The  information  was  obtained  from  3 sections  of  an  introductory  FORTRAN  programming 
class  taught  in  the  engineering  department  at  the  University  of  Washington.'  Al- 
together, 633  runs  from  54  students  working  on  4 programming  problems  were  collected 
and  analyzed.  Each  of  the  more  than  40,000  source  code  instructions  submitted  for 
processing  were  collected  and  analyzed  by  a special  purpose  program  modeled  after 
one  described  by  Nagy  and  Pennebaker  (1974).  In  addition,  another  routine  was 
developed  to  print  a special  purpose  questionnaire  on  each  output  from  each  run  to 
request  the  information  described  above. 
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• Table  1 provides  us  with  an  initial  overview  of  the  student  programming  per- 
formance across  sections  and  problems.  There  was  a chasm  between  students'  expec- 
tations regarding  their  programs  and  reality  (see  Figure  2).  After  2 runs  82% 
of  the  students  expected  to  have  completed  their  programs,  but  in  fact,  only  21% 
did.  After  6 runs,  the  comparative  percentages  were  91  and  71,  and  after  10  runs, 

100  and  87.  If  such  incredible  inaccuracy  exists  for  novice  programmers,  it  is 
not  surprising  that  professional  programmers  were  unable  to  estimate  time  and 
costs  of  developing  a program  more  accurately.  To  improve  the  accuracy  of  these 
estimations,  some  feedback  system  must  be  implemented.  System-wide  or  user- 
developed  systems  to  maintain  run  and  cost  data  might  be  helpful.  Interestingly, 
we  observed  that  those  students  who  performed  well  in  the  memory  for  programs 
experiment,  as  well  as  those  who  had  higher  scores  on  the  digit  span  test,  took 
fewer  runs  to  complete  their  programming  assignments,  (see  Table  2).  There  was 
also  an  inverse  relationship  between  performance  on  the  free  recall  learning  task 
and  the  number  of  logical  errors  reported  in  programs.  We  have  evidence  here  for 
a relationship  between  progranuiing  performance  and  human  information  processing 
ability,  albeit  complex! 

One  measure  of  student  programmer  performance  obtained  unobtrusively  by  the 
computer  was  the  number  of  runs  required  to  complete  each  programming  problem. 

A stepwise  regression  analysis  was  done  using  the  mean  runs  required  by  each  student, 
for  all  problems  as  a dependent  variable,  and  18  other  measures  obtained  from  the 
questionnaires,  the  measures  of  human  information  processing,  and  score  obtained 
in  the  course  as  independent  variables.  The  results  of  this  analysis  are  presented 
in  Table  3 with  the  independent  variables  listed  in  the  order  in  which  they  entered 
into  regression  equations. 

From  this  first  analysis,  we  see  that  score  in  class  was  not  strongly  related 
to  performance  as  measured  by  number  of  runs,  but  seems  related  more  strongly  to 
the  number  of  logical  errors  reported  for  a problem,  digit  span  score,  and  program 
preparation  time.  Notice  that  the  longer  one  spends  designing  a program,  the 
more  runs  are  required  to  complete  the  program;  but  the  more  time  coding  and  key- 
punching a program,  the  fewer  runs.  Such  a relationship  might  not  exist  for  more 
complex  problems,  or  more  experienced  prograimiers . 

Another  possible  measure  of  programmer  performance  is  the  mean  number  of 
changes  required  per  run  of  the  program.  This  measure  was  a composite  of  the 
number  of  substitutions,  deletions  and  insertions.  Table  3 shows  the  results  of 
the  stepwise  regression  analysis  using  this  measure  as  a dependent  variable.  We 
see  that  students  with  high  scores  on  the  perceptual  speed  test  tended  to  make  more 
changes  per  run  than  those  with  low  scores  on  the  test.  In  addition,  the  students 
who  wrote  larger  programs  tended  to  make  more  changes.  The  same  analysis  was 
repeated,  using  substitutions  as  the  dependent  variable,  and  a similar  set  of  variables 
enters  the  equation.  However,  the  coefficient  of  determination  was  somewhat 
higher  than  in  the  previous  analysis. 

Finally,  which  of  these  measures  allows  us  to  predict  a person's  grade  in 
an  introductory  course?  The  answer  is,  very  few.  A stepwise  regression  was  per- 
formed in  which  18  measures  of  performance,  human  information  processing,  and  data 
from  the  questionnaires  could  have  been  entered  into  the  equation.  Only  two  measures 
were  significantly  related  to  score  in  the  classroom  — mean  time  to  locate  an  error 

in  a program  and  frequency  of  syntax  errors.  We  can  compare  this  relationship 

to  that  of  GPA  and  the  score  in  the  course.  For  the  27  students  who  agreed  for  us 

to  obtain  their  overall  college  GPA,  the  correlation  between  GPA  and  course  grade 
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table  1 


Figure  2 A Comparison  of  Actual  to  Expected  Number  of  Runs  Required 
to  Complete  Computer  Problems 


TABLE  2 


Significant  correlations _b8tween  measures  of  human  Information 
processing  and  computer 'prograiiming  performance. 
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Clolul  Stepwise  9cgrcsslon  Predicting  Programihing  Performance 
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was  .58  (coefficient  of  detemination  » .34).  Therefore,  course  grade  seems 
to  measure  test- taking  ability  more  than  it  measures  programming  performance 
as  defined  here. 

The  low  correlation  between  grade  and  prograiraning  performance  may  not  imply 
that  current  methods  of  assigning  grades  are  invalid,  but,  rather,  it  may  simply 
suggest  that  programming  performance  is  difficult  to  measure  objectively.  A priori, 
we  assumed  that  "A"  students  would  be  able  to  write  correct  programs,  keypunch 
them  properly,  get  them  working  in  one  or  two  runs,  and  make  few,  if  any,  changes. 

An  alternative  hypothesis  is  that  there  are  two  types  of  ''A''  students  — those 
who  are  able  to  develop  a correct  algorithm  and  implement  it  without  making  any 
logical  changes  and  "A"  students  who  have  both  algorithm  and  implementation 
difficulties,  but  persist  until  the  problem  is  solved  correctly.  "A”  student 
type  1 would  clearly  be  the  more  desirable  programmer  for  any  organization. 

Referring  back  to  Table  3 we  can  see  the  results  of  a stepwise  regression 
analysis  where  mean  logical  errors  per  problem  was  used  as  the  dependent  measure. 

The  first  two  measures  might  have  been  expected;  viz,  the  more  runs  a person  takes, 
the  more  logical  errors.  The  next  three  measures  are  measures  of  human  information 
processing  ability  and  their  presence  in  this  equation  is  sufficiently  interesting 
to  merit  an  independent  series  of  studies.  The  relationship  between  these  measures 
of  human  information  processing  and  logical  errors  is  not  that  strong,  but  reliable 
information  which  helps  us  eliminate  logical  errors  in  computer  programs  is 
extremely  valuable.  If  such  a relationship  were  substantiated  in  further  studies, 
there  are  at  least  two  distinct  benefits.  New  programmer  selection  procedures 
could  be  developed  to  select  programmers  who  have  good  short  term  memories  and 
progranming  techniques  could  be  modified  to  minimize  short  term  memory  demands. 

In  the  stepwise  regression  just  described,  variables  were  entered  into  the 
regression  equation  according  to  their  individual  ability  to  improve  the  prediction 
of  the  dependent  variable.  A similar  analysis  can  be  done  in  which  variables  are 
forced  into  the  equation  according  to  some  predetermined  ordering.  Table  4 shows 
such  an  analysis  in  which  progranming  performance  measures  were  predicted  by  (1) 
measures  of  human  information  processing,  (2)  class  grade  for  the  quarter,  and 
(3)  the  subjective  measures  obtained  by  the  questionnaire.  The  sizes  of  the 
coefficients  of  determination  are  sufficiently  large  to  add  a sizeable  measure  of 
support  to  our  hypothesized  relationship  between  short-term  memory  ability  and 
programming  performance. 

In  closing,  our  major  findings  will  be  reiterated. 

1.  Students  took  a sizeable  amount  of  time  to  use  the  FORTRAN  language  to 
solve  simple  problems.  Particular  difficulty  was  experienced  with  input/ 
output  statements. 

2.  There  was  some  evidence  that  students  may  take  a troubleshooter  approach 
to  debugging  their  programs  — find  the  obvious  error,  correct  it,  and 
try  again. 

3.  There  was  a chasm  between  students'  expectations  of  the  time  required  to 
complete  a progranming  project  and  reality. 

4.  There  was  no  significant  correlation  between  the  number  of  runs  required 
to  complete  a programming  project  and  classroom  performance. 
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5.  Analyses  and  inspection  of  the  data  suggested  that  a relationship  may 

exist  between  logical  errors  in  programs  and  the  short-term  memory  ability 
of  programmers. 


CRITICAL  FACTORS  IN  SOFTl'lARE  DEVELOPMENT 


Scanning  any  recent  publication  in  Software  Engineering,  one  discovers  a number 
of  suggestions  for  improving  the  software  production  and  maintenance  processes. 

These  suggestions  address  a number  of  problems  which  are  presumed  to  be  critical 
to  the  production  of  software.  But  nowhere  is  there  a list  of  the  most  critical 
' factors  ranked  in  some  order  of  importance.  We  at  GE/ISP  have  recently  completed 
an  effort  to  provide  exactly  this  information  (Love  and  Fitzsimmons,  1977). 

We  designed  a three  part  survey  to  send  to  software  developers  within  GE. 

In  the  first  section,  we  presented  a list  of  100  difficulties  likely  to  be  encountered 
during  a software  development  effort  and  requested  the  respondent  to  rate  the  cri- 
. ticality  of  each  difficulty.  In  the  second  section,  we  requested  a ranking  of  six 
categories  of  difficulties  to  determine  which  were  most  critical.  Then  in  the 
final  section  we  used  a series  of  open-ended  questions  to  determine  which  difficulties 
were  considered  most  important. 

Of  the  200  surveys  distributed,  89  were  received  from  4 GE  locations.  The 
respondents,  all  software  developers,  represent  a range  of  experience  with  respect 
to  job  positions,  type  of  programming  done,  number  of  years  of  experience  in  soft- 
_ ware  development,  and  programming  languages  used.  They  also  represent  a cross  section 
of  software  developers  throughout  the  country,  with  24  from  Arlington,  VA;  37  from 
* Sunnyvale,  CA;  25  from  Syracuse,  NY;  and  3 from  Schenectady,  NY. 

Section  I 

. » 

Thirteen  of  the  100  questions  had  an  average  criticality  greater  than  or  equal 
to  5.0  (or  very  critical),  (see  Table  5).  Of  these  13,  6 were  from  the  Management 
..  Plan  subgroup,  4 from  the  Requirements  Analysis,  and  one  each  from  Environment, 

Development  System,  and  Operational  System.  No  questions  from  the  People  subgroup 
- were  above  this  break  point.  ? 


While  inspecting  the  set  of  most  critical  questions  in  each  subgroup,  we 
~ discovered  some  interesting  similarities.  For  the  Requirements  Analysis,  we  find 
that  the  most  critical  aspect  is  that  the  requirements  be  compleue  and  well  specified. 

* Also,  the  procedure  for  clarifying  the  requirements  was  considered  critical. 

’’  With  regard  to  the  Management  Plan  subgroup,  all  of  the  critical  questions 

••  involve  the  estimation  and  allocation  of  appropriate  resources  to  the  software 
development  team.  This  is  a critical  need  of  software  managers  which  is  receiving 
IT  little  attention.  While  a plethora  of  so-called  software  tools  are  being  used  to 
help  the  analyst  and  coders,  no  such  set  of  tools  is  being  used  by  software  managers. 
Also  software  developers  want  more  access  to  more  reliable  software  development 
jr  systems.  Oversimplifying,  software  developers  want  to  know  what  to  do  and  they  want 
sufficient  resources  to  do  it  ( including  calendar  time,  skilled  personnel,  and 
computers). 

The  objective  of  another  analysis  of  the  data  from  Section  I was  to  investigate 
the  relationship  among  the  100  questions.  We  essentially  asked  if  we  could  have 
employed  a small  set  of  ratings  to  obtain  the  same  information.  For  this  analysis 
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REQUIREMEHTS 

Ai^ALYSIS 


MANAGE?OT 

PLAN 


DEVELQPT-eiT 

SYSTEM 


OPERATIONAL 

SYSTEM 


ENVIRONMeiT  Q46 


Q63 


Table  5.  High  Outliers 

(Questions  Rated  ^ 5.0}_ 

Section  I 


Poor  description  of  overall  purpose  of  the  software 
Lack  of  detailed  specification  of  the  problem 
Major  aspects  of  the  requirements  not  included 
Inadequate  procedure  for  clarifying  the  requirements 


Ineffective  project  management 
Poor  allocation  and  management  of  resources 
Failure  of  management  to  provide  for  sufficient  skills  and 
talents  to  accomplish  the  task 
Underestimate  of  the  resources  required  to  accomplish  the 
task 

Underestimate  of  the  difficulties  which  arise  due  to  the 
size  of  the  project 
Overly  optimistic  completion  schedule 


Insufficient  access  to  computer 


Unreliable  development  operating  system 


Lack  of  time  for  initial  testing  of  operating  system 


ENVIRONMENT 

Q45 

DEVELOPMENT 

SYSTEM 

Q75 

Q77 

Low  Outliers 
(Questions  Rated  s 3.0) 

Section  I 


Difficult  to  obtain  general  office  supplies 


Inappropriate  identation  of  code  makes  it  more  difficult 
to  understand  • 

Variables  within  the  program  tend  to  be  global  vs.  local  I , 
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we  performed  six  separate  factor  analyses,  one  on  each  logical  subset  of  the  ratings. 
Regrettably,  we  could  not  perform  a single  analysis  on  all  100  ratings  because  1 

••  only  89  surveys  were  ultimately  returned  — far  less  than  the  300  to  500  required.  | 

I 

Briefly,  factor  analysis  is  a method  for  determining  1^  underlying  factors  \ 

from  ri  measures.  These  factors,  derived  from  the  intercorrelations  among  the  n_  | 

measures,  are  hypothetical  constructs  or  variables  that  are  assumed  to  underlie  I 

the  original  measures.  The  result  of  such  an  analysis  is  not  a verbal  description  | 

of  the  underlying  factors.  Rather,  it  is  a matrix  of  factor  loadings,  rangina  from  1 

-1.00  to  +1.00,  which  express  the  relationships  between  the  original  measures  and  the 
derived  factors.  To  the  extent  that  a measure  relates  to  a factor  it  is  said  to  i 

be  loaded  on  that  factor.  However,  understanding  and  describing  the  factor  loadings  i 

which  result  from  such  an  analysis  is  a subjective  exercise  guided  by  imprecise 
rules. 

In  our  factor  analyses  we  were  seeking  to  identify  the  primary  underlying 
factors  which  are  related,  either  positively  or  negatively,  to  the  success  of  a 
software  development  project.  The  results  of  the  factor  analysis  can  also  be  used 
to  estimate  the  criticality  of  the  hypothetical  factors  so  derived. 

Table  6 lists  the  23  factors  identified  in  the  factor  analysis,  ranging  from 
most  critical  to  least  critical.  A larger  survey  of  this  type  would  reveal  an  even 
smaller  set  of  factors. 


i 
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Section  11 

Data  from  78  surveys  were  used  to  obtain  the  total  number  and  proportion  of 
votes  for  each  paired  comparison.  These  data  were  then  used  to  obtain  (1)  the  total 
percentage  of  votes  for  each  of  the  six  variables  and  (2)  the  average  proportion 
of  votes  for  each  variable  with  standard  deviations  (see  Table  7).  The  average 
proportion  of  votes  is  an  interval  measure  which  implies  that  we  can  meaningfully 
interpret  the  differences  between  the  variables  as  well  as  the  ordering.  Clearly 
the  Requirements  Analysis  and  Development  System  are  the  critical  variables 
according  to  this  analysis.  At  the  other  extreme,  the  work  Environment  seems  un- 
important to  software  practitioners. 


Section  III 

The  final  three  questions  requested  subjective  responses  concerning  the  res- 
pondent's’’ own  difficulties  and  successes  in  past  software  development  projects. 

Responses  were  categorized  into  classes  of  difficulties  reflecting  the  six  variables 
used  in  the  imiltiple  ranking  exercise.  For  example,  both  of  the  following  responses 
were  categorized  under  "Poor  Management": 

"Management  refuses  to  make  decisions  (buck-passing,  finger-pointing}." 

"Management  refuses  to  listen  to  and  accept  ideas  of  software  developers." 

When  possible,  each  class  is  broken  down  to  show  the-major  points  mentioned  in  each.  ^ 

Following  is  a comparison  of  the  results  of  Section  II,  the. paired  comparisons  f 
measure,  and  the  responses  to  Questions  6,  7, and  8 in  Section  III  (see  Figure  3). 

This  comparison  shows  the  percentage trf  votes  (or  times  mentioned)- for  each  of  the  I 

major  components  of  the  software  development  system.  ' \ 

i 
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FACTOfl 

MEAf! 

CRITICAI 

ITY  1 

Poor  project  esticition  

. . . SAZ 

Lack  of  detailed  specification  of  the  problem  ...  

. . . 5.33 

Conflicting  and  incomplete  requirements  specifications  

. . 5.09 

Poor  reliability  of  development  system  

. . 4.81 

Difficult-to-uce  computer  systems  ...  

. . 4.78 

Difficulties  using  operational  system  

. . 4.48 

Poor  diagnostics  from  operational  system  

. . 4.47 

Ineffective  personal  ability  of  manager  . . 

. . 4.46 

Ambiguous  team  structure  

. . 4.41 

Inappropriate  structure  of  software  team  

. . 4.32 

Inadequate  training  and  review  of  team  members  

. . 4.22 

Distracting  and  uncomfortable  physical  working  environment  . . . 

. . 4.14 

Difficulties  with  operations  personnel  

. . 4.13 

Inadequate  technical  reference  material  . . 

. . 4.13 

Complexity  of  code  makes  program  difficult  to  understand  

. . 4.03 

j 

Failure  to  properly  manage  development  team  ... 

. . 3.96 

Difficult-to-work-with  co-workers  . . .'  

. . 3.93 

Requirements  difficult  to  understand  . . 

. . 3.80 

Poor  programming  style  

. . 3.67 

Poor  relationships  among  co-workers  

. . 3.65 

Unspecified  softw'are  standards 

. . 3.64 

Inadequate  software  progress  monitoring  . 

. . 3.49 

Unresponsive  support  services  .....' 

. . 3.38 

J 
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Table  7..  percentage  and  Proportion  of  Votes  for  Each  Variable 

Section  II 


PERCEMTAGE 

OF 

TOTAL  VOTES 

AVERAGE  ! 
PROPORTION  1 
OF  VOTES  1 

STANDARD 

DEVIATIONS 

Requirements  Analysis 

24 

84.4 

9.5 

Development  System 

20 

68.8 

26.7 

People 

17 

58.2 

26.5 

Operational  System 

16 

55.2 

28.8 

Management  Plan 

14 

50.0 

31.9 

Environment 

9 

32.5 

33.9 

GE  Stock 

0 

0.0 

0.4* 

♦tiQTE:  One  person  did  rank  "G£  Stock"  more  important  than  "People"  1 


The  two  categories  of  difficulties  which  were  judged  most  critical  in  Section 
I were  Management  Plan  and  Requirements  Analysis.  By  contrast,  in  Section  II  we 
found  that  Management  Plan  was  ranked  fifth  with  only  14  percent  of  the  total  votes. 

But  in  Section  III,  where  we  categorized  the  responses  into  the  appropriate  cate- 
gories, we  again  see  Management  Plan  being  mentioned  second  most  frequently.  This 
suggests  that  respondents  did  not  properly  understand  what  was  impli^  by  the  term 
Management  Plan  in  Section  II  but  did  consider  it  quite  important  when  considering 
individual  aspects  of  it. 

We  also  see  variation  in  the  relative  position  of  Requirements  Analysis. 

It  is  considered  quite  critical  in  Sections  I and  II  but,  in  the  series  of 
questions  in  Section  III,  interesting  variability  exists.  Although  Requirements 
Analysis  gets  21  percent  of  the  votes  when  asked  for  the  "most  significant  difficulties", 
it  gets  only  3 percent  when  asked  "What  aspect  should  be  changed?"  This  discrepancy 
suggests  that  even  though  software  developers  consider  the  Requirements  Analysis 
important  to  the  success  of  a project,  it  is  perceived  as  an  inherently  difficult 
problem  for  which  better  procedures  are  not  likely  to  be  developed. 

Another  variation  in  relative  positions  can  be  seen  in  the  People  subgroup. 

When  asked  "What  was  the  most  important  single  factor  contributing  to  high-quality 
software?",  respondents  listed  competent  and  cooperative  people  48  percent  of  the 
time.  However,  it  was  mentioned  less  than  half  as  many  times  when  asked  "What 
aspect  of  your  current  software  development  process  would  you  change?"  Moreover, 
the  People  were  mentioned  only  one- third  as  many  times  when  asked  for  the  "most  sig- 
nificant difficulties".  This  may  indicate  that  while  the  People  are  not  the  main 
source  of  difficulties,  competent  software  developers  are  perceived  as  the  best 
solution  for  overcoming  these  difficulties. 

We  also  see  that  the  Development  System  was  ranked  first  when  asked  for  "the 
most  significant  difficulties"  and  "What  aspect  would  you  change?",  with  32 
percent  and  42  percent  of  the  votes,  respectively.  On  the  other  hand,  the  Develop- 
ment System  was  the  "single  contributing  factor  to  high-quality  software"  only 
16  percent  of  the  time.  This  suggests  that  software  developers  are  having  a 
significant  amount  of  productivity  difficulty  with  the  Development  System,  a 
problem  which  they  feel  could  and  should  be  dealt  with. 

We  think  of  this  survey  as  a pilot  study  before  a more  sizeable  survey  effort 
is  done.  With  as  many  as  1,000  respondents,  we  could  perform  deeper  analyses  which 
might  prove  to  be  quite  revealing.  For  example,  how  different  are  the  problems 
in  organizations  developing  different  types  of  software  working  in  different  physical 
environments?  Are  there  consistent  differences  as  a function  of  the  type  of 
development  system  are  being  used?  Do  standards  really  seem  to  impact  the  attitudes 
or  performance  of  the  software  developers? 

Another  distinct  value  of  surveys  is  to  measure  changes  in  attitudes  as  a 
function  of  changes  in  operational  procedures.  Much  effort  is  currently  being 
expended  to  Impose  so-called  structured  programming  techniques  throughout  the 
software  industry,  including  GE.  Based  upon  the  results  of  this  survey,  this  type 
of  change  is  not  likely  to  have  great  impact  on  employee  attitude.  But,  to  measure 
the  true  impact  of  procedure  changes,  we  must  begin  to  collect  data  on  employee 
attitudes  and  employee  performance.  The  determinants  of  programming  performance  may 
be  quite  different  from  those  that  the  programmers  identify. 
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CONCLUSIOMS 


The  production  of  software  involves  a complex  system  which  is  not  only 
difficult  to  study  but  also  difficult  to  understand.  Because  of  the  inherent 
complexities  in  such  a system,  changing  just  one  component  in  it  cannot  be 
done  in  isolation.  Any  change  is  going  to  perpetuate  further  changes  — some 
predictable  and  some  quite  unpredicatable,  some  beneficial  and  so:ne  unbeneficial . 

In  order  to  tune  the  system  to  make  it  more  efficient  in  a predictable  fashion, 
a scientific  approach  is  essential.  The  major  components  of  the  system  need  to 
be  monitored  over  time  to  determine  the  precise  effects  of  perturbations  in  the 
overall  system.  By  properly  combining  this  monitoring  with  basic  research,  software 
production  can  be  transformed  into  a predictable  process. 

Once  the  system  is  understood  and  sufficient  monitoring  data  have  been 
scrutinized,  we  will  be  better  able  to  predict  and  control  the  current  high  costs 
associated  with  software  production  and  maintenance. 
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CGKPOTZR  SYSTOiS  ENGINEERING  MANAGEMENT  EDUCATION 

Dr.  Jofaa  H.  Maiil«7 

A«alAtant  to  thtt  Dlroctor 
Tha  Jofana  Hopkins  Univarslty 
Appllsd  PfaiTSles  Lsbozstory 


Ths  advent  of  mleroproeassors»  eoammlcatlon  satallltles,  on-llns 
syatMo  and  a wlda  variety  of  aobedded  computer  aystaia  has  Increased 
the  depth  of  technical  competence  required  to  develop  and  Integrata 
automated  systam  components.  The  trend  toward  system  development 
using  building  blocks  (modules)  possessing  "standard”  interfaces  has 
permitted  the  emerging  field  of  software  engineering  to  grow 
fxploslvely.  The  new  "software  engineer"  no  longer  requires  detailed 
knowledge  of  the  internal  logic  of  most  of  the  growing  number  of 
software  modules  he  uses  to  develop  increasingly  complex  systeu.  On 
the  other  hsnd»  computer  system  engineering  managers  need  specialized 
assistance  from  a wider  variety  of  technologists  in  this  new 
environment  to  help  than  make  decisions  such  as  selecting  a "best"  set 
of  modulest  estiaating  systam  costs,  determining  work  breakdowns, 
developing  training  programs,  and  so  forth. 

Thus,  technological  change  has  created  and  la  continuing  to  expand 
three  somewhat  new  and  separata  career  fields  in  the  computing  world: 

(1)  Coaiputer  systam  engineering  msnagenent, 

(2)  computer  system  engineering  and  neintensnea,  and 

(3)  computer  systam  resource  engineering. 

These  occupational  modifications,  together  with  newly  defined 
divisions  of  labor,  have  caused  many  of  our  traditional  education 
programs  in  the  computing  world  to  become  obsolescent,  especially  in 
the  area  of  manegsmant.  It  is  well  known  that  many  leadi^ 
universities  have  addressed  the  software  engineering  problem  and  have 
made  greet  progress  in  this  area.  The  need  for  a more  "systams 
oriented"  and  an  "engineering-type"  approach  to  building  automated 
systems  is  also  wall  known,  rozmsl  laiiverslcy-level  training  of 
computer  systam  engineering  managers,  on  the  other  hand,  is  rslativaly 
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At  The  Johns  Hopkins  Unlvorslty*  coursss  In  "conpucsr  systsas 
•nglnssring  mansgenent''  for  graduats  studants  in  Elactrlcal 
Englnaarlng  and  Con^tar  Sclenca  hsva  baan  davalopad  and  taught  for 
tha  first  tlaa  in  tha  1976-77  school  yaar.  Tha  two-saaastar  sarlaa 
has  also  baan  sumnarlzad  and  taxight  in  a 1977  suanar  sassion  contputar 
scianea  survey  coursa. ' Tha  dlstlnctiva  approach  at  Hopkins  involves  a 
blandlng  of  science  and  ■snsgsmsnt  through  modeling  and  stiidant  on- 
tha-job  exparlmantatlon.  A special  effort  is  aada  to  dlffarantiaca 
batvaan  "asnagamenc  style'*  and  'haaaagcnant  scianea".  Tha  program  la 
orlgntad  to  first  provide  tha  student  with  a global  parspactiva  of 
computer  syataa  managsmant  and  what  top-level  policymakers  desire, 
a.g.,  DdDD  5000.29.  A primer  on  managsmant  la  provided  to  show  wtot 
is  "style"  and  ahat  is  not. 

Through  various  projects  and  classroom  cxarclaas,  tha  differancas 
In  nsnagament  thinking  and  technologist  thinking  is  ezplalnad  based 
upon  tha  physiological  fact  of  tha  lateralization  of  tlie  human  brain. 
Using  this  foundation,  a "bottom  up"  approach  to  Instilling  dlsciplina 
in  tha  computer  systam  management  world  is  than  taught 

using  a "General  Procedural  Modal."  finally,  tha  student  attempts  to 
practice  what  has  baan  preached  by  waking  a real  managnent 
iayrovamant  in  hia  own  organization. 


Tha  courses  blend  real  world  managamant  with  managament  scianea, 
behavioral  science,  "pure"  aciantific  method,  englnaarlng 
mathodologlas,  and  most  important  of  all,  rnmainn  sanaa.  Tha  bottom 
Una  is  to  inereasa  student  awarenass  of  tha  managamant  world  and 
provide  tham  with  a raallatlc  methodology  for  improving  that  world  on 
a day-to-day  basis. 

This  presentation  will  provide  a summary  of  tha  techniques  used  in 
the  Johxis  Hopkins  educational  program  to  ganarata  discussion  and 
develop  new  research  initiatives  In  this  araa. 
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Computer  Syatcma  Exiglneerlng  Maxiegement  Educeclon 
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Dr.  John  H.  Henley 


Beckground 

Historical  Problems 
Current  Solutions 
Continuing  Problems 

Requlments  for  A More  Permanent  Solution 

Differentiation  Between  Policymaking  and  Practice 

Differentiation  Between  Top-Down  and  BottomrUp  Change 

Inpraved  MaaDzy  Structures  for  All 

Broadened  Knowledge  Base 

Change  Implementation  Skill  Development 

New  Research  Initiatives 

Academic  Subjects 

Computer  System  Engineering 
Project  Management 
Behavioral  Science 
Decision  Science 
Management  Science 
Scientific  Method 
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L«arn  to  Ua«  "Right-Brain"  to  Conuimlcata  With  Managament 
Ratum  to  "Laft-Braln"  vnian  Appropriata 
Promo ta  "Bottom-Up"  Changa  Consistent  With  "Top-Doun"  Policies  | 

Reap  Focus  on  Computer  Systems  as  a Functional  Axaa 

L 


A THEORY  OF  THE  MARKET  DEMAND  FOR  INFORMATION 


ANALYSIS  CENTER  SERVICES 


Th«  purpose  of  theory  is  Co  explain  observations  by  showing  that  the 
observations  are  analogous  to  or  special  cases  of  known  and  predictable 
relationships.  Just  as  the  heliocentric  theory  of  the  solar  system 
"explained"  observations  on  planet  movement  by  desuinscraclng  the  movement 
was  predicted  by  Che  known  laws  of  terrestrial  physics,  and  the  first 
theories  of  gas  behavior  "explained"  by  positing  an  analogy  between  gas 
molecules  and  solid  vibrating  spheres  obeying  the  laws  of  mechanics,  a 
theory  of  demand  for  Information  Analysis  Center  (lAC)  services  should 
explain,  or  account  for,  observations  on  demand  by  showing  they  are 
consistent  with  accepted  economic  theory.  (One  usually  prefers  to  speak 
of  accepted  theory  .-ather  chan  known  laws  in  economics.)  The  purpose 
of  this  paper,  then.  Is  to  develop  a theory  of  the  demand  for  lAC 
services  - a theory  consistent  with  accepted  economic  theory  and  consistent 
with  Che  observations  on  demand  fo?*  these  services. 


An  Information  Analysis  Center  Is  a formally  structured  organizational 
unit  specifically  but  not  necessarily  exclusively  established  for  the 
purpose  of  acquiring,  selecting,  storing,  retrieving,  evaluating,  analyzing 
and  synthesizing  a body  of  Information  and/or  data  in  a clearly  defined 
specialized  field  or  pertaining  Co  a specific  mission  with  the  Intent  of 
compiling,  digesting,  repackaging,  or  ccherwlse  organizing  and  presenting 
pertinent  Information  and/or  data  In  a fom  most  authoritative,  timely,  and 
useful  to  a society  of.  peers  and  management. 
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For  concreteness,  this  paper  will  concentrate  on  analyzing  demand 
for  two  specific  lAC  services:  providing  a handbook  and  providing 
inquiry  response.  The  model  and  results  are  generalizable,  with  some 
loss  of  traceability,  to  the  provision  of  many  other  information  services. 
The  observations  to  be  explained  may  be  cast  as  "stylized  facts,"  to 
borrow  a term  from  the  economic  growth  theory  literature.  These  stylized 
facts  are: 

1.  Low  willingness- to-pay  for  lAC  inquiry  response 
by  potential  and  actual  users. 

2.  High  elasticity  of  demand  for  inquiry  response. 

3.  Low  volume  of  service  actually  demanded  from  lAC's. 

4.  High  value  of  the  information  provided  by  the 
lAC  to  its  users. 

3.  Adverse  effects  of  handbook  sales  on  demand  for 
inquiry  response. 

Facts  1,  2,  and  3 can  all  be  considered  as  reflecting  a lackluster 
market  demand  for  lAC  inquiry  response.  Fact  4 gives  rise  to  a paradox 
(and  the  need  for  explanation)  since,  typically,  the  information  provided 
by  an  lAC  inquiry  response  service  is  very  important  to  the  customer. 
Presumably,  he  should  be  willing  to  pay  quite  a bit  to  get  the  information, 
and  his  demand  should  be  relatively  inelastic.  The  fifth  fact  presents 
no  paradox,  but  is  simply  an  observation  which  a successful  theory  should 
be  capable  of  accounting  form. 

Since  an  lAC  provides  information  services,  it  is  important  to 
recognize  that  it  is  the  value  of  their  service,  not  the  value  of  the 
information,  which  determines  the  economic  viability  of  an  lAC.  This 
distinction  is  important  as  long  as  the  lAC  does  not  occupy  a monopolist’s 
position  with  respect  to  any  types  of  information.  To  see  this,  consider 
the  analogous  problem*vof  valuing  the  services  of  a surgeon  who  performs 
lifesaving  operations.  Is  the  value  of  the  surgery  the  value  of  the  life 
saved?  Yes*.  Is  the  value  of  the  surgeon's  service  the  value  of  the 
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life  saved?  No!  This  is  simply  because  there  exist  a large  number  of 
surgeons  with  comparable  skill.  The  lifesaving  surgery  could  be  provided 
by  a large  number  of  service  (surgery)  providers.  The  value,  to  the 
patient,  of  that  surgeon's  service  is  the  cost  of  another  equally  skilled 
surgeon's  service.  If  all  surgeons  (assumed  equally  capable)  charged 
$1000  for  a particular  operation,  the  value  of  any  one  surgeon's  service 
is  only  $1000,  since  that  is  the  cost  of  acquiring  a perfect  substitute. 

The  value  of  the  service  of  a particular  supplier  and  the  value  of  the 
service  supplied  are  vastly  different  concepts.  The  same  point  can  be 
made,  perhaps  more  clearly,  with  a physical  cod^odity  rather  than  with 
a service.  Consider  the  commodity,  aspirin.  In  Figure  5,  asstime  D is 
the  market  demand  for  bottles  of  aspirin.  If  aspirin  were  supplied  by  a 
perfectly  discriminating  monopolist,  he  could  charge  each  consumer  the 
total  of  what  each  consumer  values  the  aspirin.  Thus,  the  first  bottle 
could  be  sold  at  P^,  the  second  at  P^,  etc.  The  monopolist  is  able  to 
sell  his  provision  of  the  good  at  the  value  of  the  good  itself  to  each 
particular  buyer.  This  situation  can  be  contrasted  with  a competitive 
market  in  which  many  firms  provide  a homogeneous  good  (each  firm’s  product 
is  indistinguishable  from  that  of  any  other  firm),  and  no  one  of  them 
provides  a significant  amount  of  the  total.  Let  P*  be  the  price  which 
prevails  under  this  case.  Each  supplier  can  sell  his  provision  of  the  good 
at  a price  no  higher  than  other  suppliers  sell  their  provision.  For  to 
set  a higher  price  would  be  to  lose  all  sales,  since  no  buyer  will 
willingly  buy  the  same  good  at  a higher  price.  Thus,  the  price  which  can 
be  charged  in  the  competitive  market  does  not  depend  on  the  value  of  the 
good  itself,  but  on  the  prices  other  suppliers  set. 

The  upshot  of  the  foregoing  is  that  insofar  as  lAC's  are  not  the 
sole  providers  of  information,  the  prices  buyers  of  Information  are  willing 
to  pay  the  lAC  do  not  depend  on  the  intrinsic  value  of  information,  .but 
on  the  prices  other  information  suppliers  have  established.  The  point  is 
stressed  here  because  it  is  so  easy  to  lapse  into  the  mistaken  notion  that 
the  value  of  an  lAC  is  the  value  of  the  Information  it  provides.  Rather, 
its  value  is  the  cost  of  the  best  alternative  information  source. 
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Only  in  Che  instance  Chat  an  lAC  provides  Cotaily  unique  informatlun  can 
the  lAC's  value  be  measured  by  the  value  of  the  information  (i.e.,  the 
lAC  is  a monopolist). 

As  a first  approximation,  imagine  that  all  questions  ask  for  the 
numerical  value  of  some  parameter.  Parameters  might  include  physical 
constants,  economic  values,  production  levels,  dates,  etc.  Assume  a 
measurable  and  known  cost  to  the  user  of  acquiring  each  item  of  information. 
Likewise,  assume  a measurable  and  known  monetary  benefit  to  the  user 
associated  with  each  item.  The  rational  user  will  determine  the  net 
benefit  of  each  item  (benefit  less  cost)  and  purchase  the  information 
item  as  long  as  net  benefits  are  positive.  The  whole  set  of  information 
items  may  be  arranged  according  to  diminishing  net  benefit,  as  in  Figure  1. 
The  curve  is  interpreted  as  follows:  the  information  item  yields 

net  dollar  benefits  of  v^.  q*  is  the  quantity  purchased  by  the  rational 
user  since  each  unit  up  to  q*  adds  more  to  net  benefits  and  each  unit 
beyond  q*  detracts  from  net  benefits. 

Some  reflection  will  reveal  that  the  graph  of  benefits  and  the  graph 
of  costa  of  each  information  item  (arranged  along  the  horizontal  axis  in 
the  same  order  as  in  Figure  1)  will  not,  in  general,  be  representable  by 
a smooth  curve.  For  the  (gross)  benefit  of  any  item  may  differ  markedly 
from  the  (gross)  benefit  of  an  adjoining  item,  and  likewise  for  costs. 

All  that  is  known  for  sure  is  that  the  differences  in  benefits  and  costs 
decline  as  one  moves  to  the  right  in  Figure  1. 

Figure  2 illustrates  benefit  and  cost  curves  which  could  give  rise 

to  a net  benefit  curve  like  Figure  1.  So  far,  no  mention  has  been  made 

of  the  lAC.  In  particular,  the  cost  curve  in  Figure  2 (and  thus  implicit 

in  Figure  1)  represent  the  costs  in  the  absence  of  the  LAC.  Now  suppose 

the  lAC  can  provide  information  to  the  user  at  a constant  unit  cost, 

say  c.  If  c were  less  than  - meaning  that  the  lAC  could  alwys  provide 

Information  at  a lower  cost  than  any  other  information  service  (where 

providing  information  to  oneself  is  one  of  those  "other"  information 

services),  the  user  would  purchase  all  information  from  the  lAC  (case  C^) . 

On  the  other  hand,  if  c were  greater  than  C„,  the  user  would  purchase  no 
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Informacion  from  the  lAC  (case  C2) . The  first  of  these  cases  Ls  not 
realistic  and  will  be  ignored.  The  second  case  is  of  limited  interest, 
and  will  be  considered  an  extension  of  the  final  case,  where  ca  falls 
between  and  (case  • Here,  the  lAC  acts  as  a peak  shaver  - the 
user  will  substitute  lAC  service  for  some  other  service  whenever  a cost 
"spike"  (see  the  figure)  is  cut  by  the  lAC  cost,  c. 

The  introduction  of  an  lAC  can  have  two  effects.  First,  it  can 
cause  a user  to  substitute  lAC  service  for  other  service  because  of  lower 
cost,  and  it  can  cause  an  increase  in  the  amount  of  information  denanded, 
because  benefits  may  exceed  c beyond  q*. 

Clearly,  for  any  user,  the  lower  the  value  of  c the  greater  is  his 
use  of  the  lAC.  Focusing  simply  on  the  number  of  inquiries  directed  to 
the  lAC,  a demand  curve  for  lAC  inquiry  service  can  be  derived  from 
Figure  2.  By  starting  at  c ■ and  successively  lowering  c,  the  number 
of  inquiries  the  user  would  direct  to  the  lAC  can  be  noted  at  each  c. 

In  Figure  3 the  locus  of  such  points  is  plotted  as  curve  D. 

Now  consider  the  effect  that  an  lAC  handbook  would  have  on  the 

demand  curve  in  Figure  3.  By  increasing  the  ready  accessibility  of  a 

large  amount  of  information,  the  user's  oimership  of  a handbook  would 

decrease  the  cost  of  obtaining  some  information.  Consequently,  many 

of  the  peaks  on  the  cost  curve  in  Figure  2 would  be  lowered,  resulting 

in  a diminished  usage  of  the  lAC  inquiry  response  service  at  any  level 

of  c.  This  has  the  effect  of  shifting  the  curve  in  Figure  3 to  the  left. 

The  new  curve  is  represented  as  D„,  the  demand  for  inquiry  service 

n 

given  access  to  a substitute  handbook. 

The  market  demand  for  an  lAC's  inquiry  response  service  is  the  sum 
of  the  demands  of  each  user.  Graphically,  the  summing  is  accomplished 
by  adding  together  each  user's  denand  curve  horizontally.  The  process 
is  illustrated  for  two  users  in  Figure  4,  using  linear  relations  for 
clarity.  Kinks,  such  as  the  one  in  the  rightmost  quadrant,  would  be 
smoothed  out  if  many  curves  were  added  together. 


A Simplified  Development 


The  exposition  of  our  model  is  greatly  simplified  if  we  agree  to 

work  with  smooth  curves  rather  than  the  jagged  and  kinked  relationships 

in  Figures  2 and  4.  In  this  spirit.  Figure  5 represents  the  user's 

demand  for  Information.  The  height  of  the  curve  above  any  point  on 

the  horizontal  axis  tells  the  benefit,  or  value,  of  that  unit  of 

information  (response  to  an  inquiry)  to  a user.  In  Figure  6,  S is  the 

marginal-cost-of- information  curve,  excluding  consideration  of  the  lAC. 

The  height  of  the  curve  at  any  level  of  inquiries  cells  the  incremental 

cost  to  the  set  of  users  of  one  more  inquiry  response.  The  curve  slopes 

upward  to  the  right,  indicating  that  increasing  amounts  of  information 

became  more  and  more  costly.  "Cost,"  in  this  context,  refers  to  either 

actual  dollar  expenditures  by  users  or  the  dollar  value  of  expenditures 

of  users'  own  time  and  resources.  Note  Chat  our  model  divides  all 

information  sources  into  two  sectors:  The  lAC  and  all  others.  A includes 

Che  latter,  but  excludes  the  former.  Implicit  in  the  definition  of  S 

is  economic  efficiency.  By  this  is  meant  that  Che  height  of  S represents 

Che  minimum  cost  to  users  of  acquiring  that  incremental  unit  of  information. 

Thus,  S is  constructed  assuming  the  least  costly  source  is  used  for 

supplying  each  unit  of  information  service.  S„  is  the  same  supply  curve, 

n 

but  shifted  downward  in  the  event  the  lAC  provides  a user  handbook. 

The  handbook,  of  course,  lowers  user  information  costs  by  reducing  the 

time  required  to  locate  some  information  items.  This  is  reflected  in  Che 

lower  chan  otherwise  marginal  cost  curve,  S„.  S and  S„,  coupled  with  D 

n n 

of  Figure  5,  become  a special  case  of  supply-demand  analysis. 

Let  P be  Che  unit  price  of  LAC  inquiry  response.  First,  in  the 
absence  of  an  lAC  handbook,  how  much  information  will  Che  user  purchase 
altogether,  and  liow  much  from  die  lAC's  inquiry  response  service?  To 
answer  this,  begin  at  quantity  0 and  move  incrementally  to  the  right. 

Since,  between  0 and  Q^,  the  demand  curve  (hence  wllllngness-co-pay) 
exceeds  the  cost,  at  least  will  be  purchased.  Will  the  units  be 
purchased  from  the  lAC's  inquiry  response  service?  Clearly  not,  since 
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each  unit  coses  less  on  S chan  on  P.  However,  between  Q,  and  Q. , 

13’  lAC 

Is  less  chan  S,  and  is  also  less  Chan  D.  Therefore,  Che  user  will 

purchase  but  from  che  lAC.  To  che  rlghc  of  willingness- 

Co-pay  is  less  Chan  cosc  (i.u.,  D curve  Is  below  P line),  so  no  purchase 

Is  made.  Thus,  of  a cocal  purchase  of  Q^,  only  is  purchased  from 

the  lAC.  The  remaining  unlcs  are  purchased  from  che  least  expensive 

"other"  suppliers  of  the  Information  service.  As  noted  above,  this  can 

Include  self-supply.  At  this  point  che  effect  of  che  handbook  Is  evident. 

tflch  Sg  che  supply  curve,  only  (In  addition  Co  che  handbook)  would 

be  purchased  from  the  lAC.  The  model  suggests  chat  che  effect  of  a 

handbook  Is  Co  reduce  Inquiries,  as  has  been  apparently  observed.  Figure  6 

can  be  used  Co  derive  che  demand  curve  for  lAC  Inquiry  response.  This  Is 

done  by  varying  P,  and  noting  che  change  In  quantity  of  Inquiry  response 

demanded.  Thus,  using  S as  che  supply  curve.  If  P were  slightly  higher, 

less  than  would  be  demanded.  Indeed,  quantity  demanded  goes  Co 

0 at  a price  equal  Co  che  Intersection  of  S and  D.  Some  reflection 

Indicates  chat  che  demand  for  Inquiry  response  Is  given  by  che  difference 

between  D and  S,  as  long  as  D ->  S Is  positive.  Figure  7 Illustrates  this 

demand,  denoted  0^^.  Note  It  Is  uniformly  lower  chan  0,  reflecting  che 

point  made  earlier  about  che  difference  between  che  value  of  Information 

and  the  value  of  an  lAC's  providing  Information.  Also,  since  che 

observed  point  on  0 will  be  rlghc  of  che  observed  point  on  the 

demand  for  Inquiry  response  will  be  more  elastic  chan  che  demand  for 

Information..  Finally,  note  chat  if  S„  applies  In  Figure  6,  D„  applies 

cl  B 

In  Figure  7. 

This  model  of  market  demand  for  lAC  Inquiry  response,  stimmarlzed 
by  Figures  2,  3,  4 or  5,  6,  7 (for  che  "smooch”  case),  explains  che 
observations  sec  out  earlier.  In  particular,  the  concept  of  a user’s 
own  suppJy  of  InfocnuitLon  curve  pennies  the  |>arudox  of  high  Infonnaclon 
value  - low  demand  for  Information  service  to  be  resolved.  While  che 
Information  has  high  value,  the  providing  of  the  Information  has  value 
limited  by  che  relative  cost  to  Che  user  of  competing  services,  as  reflected 
In  che  users'  supply  curve.  Further,  Ic  Is  this  curve  which  leads  Co  che 
difference  In  elasticities  between  cocal  Information  demand  and  lAC 
Inquiry  response  demand. 
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Consider  a cosc-beneflt  analysis  (CDA)  of  a cwo  service  (handbook 
and  Inquiry  response)  lAC.  Ac  Issue  Is  whether  Che  existence  of  the  lAC 
can  be  justified  on  economic  grounds,  and  which  services  it  should  provide. 
Simply,  does  the  value  of  all  Che  benefits  provided  by  the  lAC  exceed  the 
costs  of  running  the  lAC?  The  problems  traditionally  encountered  in  such 
economic  studies  of  Information  services  are  of  three  types: 

1.  Identifying  all,  but  only,  the  legitimate  costs 
and  benefits  of  the  lAC; 

2.  Constructing  conceptual  measurement  schemes; 

3.  Gathering  Che  appropriate  data. 

We  hope  to  show  chat.  In  an  admittedly  limited  way,  our  theory  helps 
mitigate  these  problems.  Including  Che  last,  by  suggesting  that  under 
certain  circumstances  limited  data  Is  sufficient. 

CBA  of  an  lAC  demands  the  introduction  of  several  additional 
concepts:  the  social  demand  curve  and  the  average  cost  curve.  The 
social  demand  curve  shows  society's  willingness  to  pay  for  each 
incremental  quantity  of  Infocmatlon,  where  society  Includes  Che  user. 

The  importance  of  the  concept  stems  from  the  recognition  that  there 
potentially  exist  positive  externalities  In  the  consumption  of  Information 
l.e. , not  only  do  the  direct  users  benefit  but,  through  processes  not 
completely  understood,  others  eventually  benefit  as  well.  Thus,  Che  social 
demand  curve  is  always  at  least  as  great  as  the  users’  demand  curve. 

Figure  8 Illustrates  these  demand  curves.  The  reader  should  bear  In  mind 
that  Che  very  existence  of  benefits  over  and  above  Chose  accruing  to  the 
users  Ls  u matter  of  conjecture.  No  fLcm  evidence  of  tliese  heneflts  li.is 
yet  been  adduced.  Without  such  evidence,  the  shape  of  the  social  demand 
curve  Is  also  a matter  of  conjecture.  But  standard  economics  suggests 
these  external  benefits  be  represented  as  diminishing  with  increases  in 
total  information  supplied.  0 Is  the  market  demand  curve,  0^  the  social 
deotand  curve. 
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Figure  9A  illustrates  the  total  cost  curves  for  an  lAC.  TC  represents 

the  total  costs  of  an  lAC  doing  inquiry  response  alone,  and  TC  adds  to 

H 


that  the  (assumed)  fixed  costs  of  publishing  a handbook.  Assume  the  LAC 
information  bases  for  the  inquiry  response  and  for  the  handbook  are 
identical.  Also,  assume  constant  returns  to  scale  in  Inquiry  response. 
Figure  9B  displays  the  average  cost  curves  derived  from  the  total  cost 
curves. 

The  complete  model  for  the  CBA  of  the  two-service  lAC  is  presented 
in  Figure  10.  We  assume  the  unit  price  of  inquiry  response  at  the  lAC 
is  set  at  P.  P may  be  set  at  marginal  cost  or  any  other  value.  The 
following  analysis  may  be  carried  out  with  P a variable.  If  desired. 

This  would  be  appropriate  if  the  desire  were  to  optimize  some  value,  such 
as  net  social  benefits,  lAC  net  income,  etc.  To  simplify  exposition, 
however,  we  maintain  P a constant  here.  Let  us  use  the  model  to  analyze 
the  relative  merits  of  four  alternative  situations: 


1.  No  lAC 

2.  LAC  with  only  inquiry  response 

3.  lAC  with  only  handbook 

4.  lAC  with  both  inquiry  response  and  handbook. 

Let  us  first  analyze  alternative  iH.  Using  economic  surplus  as  the  value 
measure,  the  net  benefits  associated  with  the  first  possibility  are  area 
ABCE.  This  is  the  total  surplus  of  social  willingness  to  pay  over  costs 
incurred  when  Q'  units  of  information  are  obtained.  Q* , of  course,  is 
the  value  determined  by  the  intersection  of  S and  D. 

Now  posit  the  existence  of  an  LAG  offering  only  inquiry  response, 
priced  at  P per  unit.  Altogether  Q*'  units  of  information  will  be  obtained, 
Q''  - Q'''  from  the  lAC.  The  net  benefits**  of  this  alternative  are 
ALKE  + LCQ'  ’Q"  ' - ZYQ'  ’Q"  ’ . 


By  this  meant  that  any  change  in  output  causes  a proportional  change  in 
costs,  e.g.,  doubling  output  doubles  costs. 

** 

Since  P may  be  arbitrary,  the  area  under  P is  not  necessarily  a measure 
of  real  cost.  Thus,  gross  benefits  and  gross  costs  correcr'y  enter  the 


The  nec  benefits  associated  with  alternative  #3  are  AMNF  - T,  where 
T is  the  total  cost  of  publishing  the  handbook  alone,  and  Q is  the  quantity 
of  itrfomatlon  obtained. 

The  net  benefits  of  alternative  #4  are  ABJF  + BGQ”Q'  - WXQ'’Q'. 

Clearly,  the  existence  of  the  lAC  is  justified  if  any  of  alternatives 
2,  3,  or  4 have  greater  net  benefits  than  1.  The  preferred  configuration 
for  the  lAC  is  given  by  the  alternative  (2,  3,  or  4)  with  greatest  net 
benefits.  As  alght  be  expected,  the  issue  can  only  be  resolved  by 
eaplricsl  analysis,  axid  then  on  a case-by-case  basis. 


An  econoalc  model  has  been  constructed  which  appears  to  successfully 
account  for  a number  of  qualitative  observations  on  the  characteristics 
of  the  market  demand  for  LAC  services.  The  model  shows  that  in  the  presence 
of  competing  information  delivery  systems,  the  lAC  cannot  expect  the  desnard 
for  its  services  to  mirror  the  demand  for  information.  The  model  provides 
a useful  structure  for  analyzing  various  economic  issues  pertaining  to 
lag's,  including  viability,  mix  of  services,  and  pricing.  The  paper 
concludes  with  a conceptual  approach  to  cost-benefit  analysis  using  the 
model. 
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A FORUM  APPROACH  TO  SOFTWARE  DEVELOPMENT  - 
THE  NATIONAL  FORUM  ON  SCIENTIFIC  AND  TECHNICAL  COMMUNICATION 

This  pacer  will  suitmarize  the  activities  and  findings  of  the 
National , Forum  on  Scientific  and  Technical  Communication,  a fifteen 
month  study  supported  by  the  National  Science  Foundation,  Division  of 
Science  Information  and  conducted  by  the  George  Washington  University, 
Science  Communication  Division.  The  Forum's  relevance  to  software  life 
cycle  management  lies  in  its:  1)  evaluation  of  the  current  delivery 
systems  for  scientific  and  technical  information  (STI);  2)  reliance 
on  the  users  of  STI  for  their  suggested  improvements  to  the  delivery 
systems;  3)  consideration  to  the  current  providers  of  STI  and  their 
problems;  and,  4)  the  development  of  a model  of  an  improved  scientific 
and  technical  communication  system  and  its  presentation  to  management  - 
Yfho  in  this  case  are  decentralized,,  autonomous  and  specialized.  The 
report  will  focus  on  the:  1]  methodology  of  the  Forum;  2)  findings; 

3)  advantages  and  disadvantages  of  a Forum  approach;  and,  4)  future 
work  of  the  Forum. 

The  availability  and  delivery  of  scientific  and  technical  infor- 
mation in  the  United  States  is  related  to  the  quality  of  scientific 
research,  the  economy  (as  a result  of  technological  developments),  the 
competitive  position  internationally  and  the  national  security.  The 
legislation  that  established  the  National  Science  Foundation  included 
provisions  for  improving  the  interchange  of  scientific  information 
among  scientists  in  the  United  States  and  foreign  countries.  This 


authority  was  considerably  expanded  in  1958  by  direction  to  the  Foun- 
dation to  ► provide  or  arrange  for  the  provision  of  indexing, 
abstracting,  translating,  and  other  services  leading  to  more  effective 
dissemination  of  scientific  information”  and  . . undertake  programs 
to  develop  new  or  Improved  methods^  including  mechanized  systems,  for 
making  scientific  information  avaiTabTe." 

The  Division  of  Science  Information  (DSI),  formerly  the  Office  of 
Science  Information  Service,,  implemented  this  mandate  through  activities 
designed  to  improve  the  means  for  disseminating  scientific  information. 
Until  1974-  this  effort  was  toward  strengthening  and  expanding  science 
information  activities  of  the  professional  scientific  and  technical 
societies*  Since  1974,  DSI  funding  has  been  for  projects  designed  to 
advance  the  state  of  the  art  in  science  connunication  by  producing 
general izable  results  that  can  help  improve  methods  of  acquiring, 
transmitting,  retrieving  and  using  information  in  any  scientific  or 
technical  area.  However,  in  recent  years,  DST  has  undergone  budget 
reduction  from  S14  million  to  $5  million  and  staff  decreases  from 
70  to  22. 

During  the  same  time  period,  private  for-profit  organizations 
developed  to  provide  specialized  services  of  acquiring,  transmitting, 
retrieving  and  delivering  scientific  and  technical  information,  off- 
the-shelf  or  customized.  These  services  are  supportive  to  or  central 
components  of  the  present  categores  of  STI.  These  categories  are: 

1)  the  discipline-oriented  systems  of  the  professional  societies  - 
each  concentrating  on  the  systematic  organization  of  knowledge  in  a 


particular  domain  of  basic  science;  2)  the  misSion-oriented  informa-tion 
systems  of  the  federal  agencies,  e.g.  for  arstronautics  in  NASA,  for 
atomic  energy  in  ERDA,  for  medicine  in  the  National  Library  of  Medicine; 

3)  the  specialized  information  activities  of  private  institutions  and  of 
industry  such  as  special  libraries,  information  analysis  centers,  indexing 
and  abstracting  companies,  data  base  services,  etc.  and,  4)  the  information 
services  that  are  maintained  by  colleges  and  universities. 

An  indication-  of  the  size  of  the  nation's  scientific  and  technical 
information  systems  is  shovm  by  the  measure  of  total  resource  expenditures. 
From  1960  to  1974,  the  GNP  grew  177  percent,  R & 0 funding  increased 
136  percent  and  scientific  and  technical  communication  resource  ex- 
penditures increased  substantially  by  an  estimated  323  percent.  Total 
conmunication  resource  expenditures  were  estimated  at  $2.0  billion  in 
1960  and  $8.5  billion  in  1974. 

Ouring-  this  period  of  growth  and  expansion,  numerous  studies 
assessed  the  quality,  quantity  and  direction  of  the  scientific  and  technical 
coimunication  systems.  In  most  cases,  policymaking  recotnnendations 
resulting  from  these  studies  were  virtually  ignored.  Meanwhile,  signi- 
ficant changes  have  taken  place  in  STI  activities,  which  are  indicative 
perhaos  of  greatef  change  in  the  future.  Recognition  of  these  changes 
has  been  reflected  in  the  National  Science  and  Technology  Policy, 
Organization  and  Priority  Act  of  1976  (PL  94-282).  This  Act  established 
the  Office  of  Science  and  Technology  Policy  (OSTP)  directed  by  the 
President's  Science  Advisor  and  assisted  by  the  President's  Science 
Advisory  Conmittee  (PSAC),  one  of  whose  members  to  be  an  expert  in 


Information  dis semi nati off.  The  Act  also  assigns  to  PSAC  the  task  of 
conducting  a survey  of  federal  science  and  technology  Including  the 
consideration  of  Improvements  for  handling  scientific  and  technical 
Information  In  existing  federal  systems  and  In  the  private  sector. 

Among  the  changes  which  are  significant  to  such  a survey  are: 

T.  Users  - One  estimate  Indicates  that  there  will  be  a 20-30t  Increase 
In  the  number  of  scientists  In  the  next  decade.  In  addition,  business. 

Industry  and  governments  are  Important  secondary  users  of  STT.  Public 
concerns  related:  to  land-use  planning,  supersonic  transports,  ecology, 
poTlutloff  and  the  quality  of  life  are  engaging  the  time  of  private 
citizens,  organized  public  and  private  groups,  and  state,  local  and 
federal  government  officials-  Discussion  of  public  policy  Issues  such 
as  these  requires  scientific  and  technical  information  accessible  to 
a scientist  In  a specialized  discipline  or  to  one  who  doing  Inter- 
dlsctpTfnary  wort  or  to  a declslon-maker/pol  ley-maker  or  to  a non- 
scientist  cltfzen- 

Z,  Providers  - The  expansion  and  diversification  of  the  "providers" 
of  scientific  and  technical  Information  to  general  groups  of  producers 
Cgena*ators,  wholesalers,  search  and  sell),  providers  (retailers,  search  j 

and  seTT),  and  Intermediaries  (librarians.  Information  resource  specialists), 
has  generated  new  Industries  and  created  new  professional  activities. 

3.  Requirements  - The  value  of  STI  solely  for  validation  of  research  j 

findings  and  minimization  of  rept-ltive  efforts  has  been  expanded  to 
require  STI  for  planning,  policy-making,  decision-making  and  the  , 

allocation  of  resources.  However,  cross-disciplinary  Information  systems 

/ 
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integrating  the  diverse  social  and  technical  information  bearing  on 
problems  such  as  the  environment  or  energy  have  not  been  developed. 

4.  Technologies  - The  capabilities  inherent  in  data  banks,  abstracting, 
indexing  and  retrieval  systems  and  expanded  use  of  terminals  permit 
increased  access,  inquiries  which  range  from  specific  and  detailed  to 
general  and  macro-oriented  and  the  ability  to  store  massive  amounts  of 
information  in  large  data  Banks.  It  was  estimated  that  in  1977,  there 
are  more  than  1,500,000  on-line  computer  terminals  installed  permitting 
wider  access  and  alternatives  for  user-oriented  formats. 

5.  Policy  - In  recent  years,  there  have  been  changes  in  federal  policy 
in  the  areas  of  page  charges,  research  direction,  mailing  rates, 
dissemination  through  MTIS  and  SSIE  and  information  analysis  centers. 

At  the  same  time  private  industry  has  assumed  new  roles  as  providers 

of  information. 

6.  Legislation  - Recent  copyright  legislation  may  resolve  the  problems 
concerning  prooerty  and  conmerciaT  rights  as.  affected  by  photocopying 
and  networking.  Anti -trust  laws  which  have  the  result  of  prohibiting 
cooperative  arrangements  among  commercial  organizations  may  help  or 
hinder  the  user  of  STI  services.  The  application  of  the  Freedom  of 
Information  Act  to  publicly  financed  government  data  bases  continues 

to  be  confusing. 

7.  International  STI  - In  recent  years,  a strong  international  scientific 
and  technical  communication  system  has  evolved.  International  infor- 
mation transfer  has  added  new  dimensions  to  STI  activities. 

These  factors  serve  to  create:  1)  a varied  population  of  users:  2) 
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an  array  of  new  solutions  to  old  proSlems;  3)  an  array  of  new  problems, 
many  of  which  have  not  been  defined;  4-)  an  emerging  population  of  for- 
profit  providers  of  scientific  and  technical  Information;  5)  Increasing 
complexity  concerning  the  location,  availability  and  accessibility  of 
Information;  6)  blurring  of  the  roles  of  the  public  and  private  sector 
responsi bill ties j and,  7)  economic  barriers  to  the  access  of  STI. 

Tit  order  to  evatuate  the  adequacy  of  the  present  scientific  and 
technical  connunlcatlon  systems  and  to  plan  for  the  future,  the  OSI 
contracted  with  the  George  Washington  University  to  explore  critical 
Issues  facing  scientific  and  technical  Information  activities  In  the 
Untted  States,  emphasizing  the  prospective  roles  of  the  public  and 
private  sectors.  Early  planning  for  the  project  established  guidelines 
fbr  subsequent  actlvltlesr 

T.  Focus  on  the  users  of  STI  and  their  determination  of  adequacies/ 
Inadequacies  In  existing  STI  delivery  systems. 

2-  Define  users  broadly,,  rather  then  equating  users  with  research 
scientists. 

3.  Conduct  most  of  the  data  collection  outside  of  Washington  O.C. 

4.  Treat  STI  as  a whole  not  by  discipline,  category  of  user  or  medium. 

5.  Rely  on  »«rkshop  approaches  to  solicit  Input. 

6.  Conduct  conferences  In  "neutral"  locations  such  as  science  museums. 

The  general  methodology  for  the  project  Included  three  conferences, 

each  oriented  towards  a different  group  Involved  In  scientific  and 
technical  communication  systans,  as  shown  In  Figure  1.  Conference 
planning  Involved  working  with  the  key  professional  representing  users. 
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Conference  I (Nov.  17.  T977  - Boston) 


Conference  II  (Feb.  18.  1977  San  Franci s 'o ' 


Figure  1 : NATIONAL  FORUM  ON  SCIENTIFIC  AND  TECHNICAL  COMMUNICATION 
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then,  providers  and  finally  policymakers.  The  first  conference,  the 
Users'  Perspective  Conference,  focused  on  the  user  of  scientific  and 
technical  Information  by  addressing  the  question,  "Are  the  users  of  scientific 
and  technical  Information  served  adequately?"  Professionals  from  public 
and.  private  organization^,  public  Interest  groups  and  the  press  came 
together  on  November  11  > 197S  at  the  Boston  Museum  of  Science  to  discuss 
their  viewpoints  and  perceptions  about  the  availability  and  utilization 
of  scientific  Information.  The  second  Conference,  the  Providers' 

Perspective  Conference  was  conducted  at  the  California  Academy  of  Science 
on  February  18,.  1977.  It  focused  on  the  providers  and/or  Intermediaries 
of  scientific  and  technical  information  by  addressing  the  question, 

"Are  the  providers  of  scientific  and  technical  Information  adequately 
meeting  the  needs  of  the  user?"  On  June  1977r  at  the  Smithsonian 
Institution*  168  participants  at  the  Policymakers'  Perspective  Conference 
addressed  the  question:  ''Are  there  needs  for  public  policies* In  the  area 
of  scientific  and  technical  communication,  and  If  so,  what  are  the  options?" 

Participants  at  each  Conference  received,  prior  to  the  Conference, 
a background  paper,  a working  vocabulary  and  a questlonaire  designed  to 
stimulate  workshop  discussions.  The  collected  questlonalres  and  taped 
workshop  discussions  provided  the  record  of  all  Conferences. 

Users*  Perspective  Conference 

The  Users'  Perspective  Conference  was  opened  with  a keynote  address 
describing  the  changing  context  of  STI.  This  was  followed  by  four 
workshops  constituting  Session  I.  The  material  developed  In  the  Session 
I workshops  wes  surnnarized  and  presented  to  the  assembled  Conference. 
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The  objective  of  Session  I was  to  determine  the  critical  problems 
and  a ranking  of  Importance  of  these  problems,  as  perceived  by  Conference 
participants  from  their  work  activities  In  laboratories,  research  de- 
partments, political  processes  and  other  situations  or  environments- where 
Conference  participants  use  STI.  Workshop  size  ranged  from  twenty  to 
twenty-five  people.  The  Intent  of  these  workshops  was  to  stimulate  thinking 
and  Identify  problem  relationships  and  subsets  of  problems  from  different 
persoectlves.  It  was  recognized  that  the  problems  of  greatest  significance 
to  the  output  of  the  Conference  were  those  which  could  be  assisted  by 

y 

public  policy  action.  However,  the  relationships  between  problem  and 
policy  was  not  always  obvious,  so  policy  action  as  a restraint  was  not 
placed  on  the  discussion.  Each  participant  was  asked  to  bring  to  the 
Conference  e completed  questional re  on  which  they  Identified  two  critical 
problems  they  encountered  and  potential  public  policy  actions  to  eliminate 
these  problems. 

The  role  of  the  presider  at  each  workshop  was:  1)  Provide  Intro- 
ductory remarks  to  the  discussion;  2)  Guide  the  discussion  toward  the 
desired  output  - seven  critical  problems;  3)  Develop  a suomary  of  the 
workshop  activities  with  the  workshop  assistant;  and,  4)  Present  a 
fifteen  minute  summary  to  the  assembled  Conference. 

The  objective  of  Session  II  was  to  determine  the  responses,  priorities, 
perceptions  and  recommendations  by  sector,  to  the  problems  and  rankings 
developed  In  the  workshops  of  Session  I,  and  presented  In  the  summarization 
to  the  assembled  Conference.  Session  II  workshops  were  organized  by 
Industrial  participants,  academic  participants,  government  participants. 
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citizen  groups  and  media.  It  was  anticipated  that  the  special  needs, 
priority  problem  areas » relationships  among  problems  and  suggested  public 
policy  actions,  perceived  by  sector,,  would  provide  useful  comparative 
data. 

The  problems  and  Issues  raised  In  the  sorkshop  discussions  may  be 
categorized  as  those  relating  to: 

K Definitions  and  Meed  for  Study 

Z.  Locating  and  Accessing  Needed  Information 

3.  Scientific  and  Technical  Information  Base 

4.  Costs 

5.  Olssairfnatfon  and  Delivery 

6^  Publlc/Private  Interface 

7.  New  Functional  Responsibilities 
These  are  sumnartzed  In  Figure  2. 

Providers*  Perspective  Conference 

The  Providers'  Perspective  Conference  was  opened  by  a series  of  talks 
designed  to  bridge  the  gao  between  the  Users'  Perspective  Conference 
and  the  Providers'  Perspective  Conference.  Significant  problems  and/or 
Issues  on  which  there  was  consensus  at  the  Users'  Perspective  Conference 
were  developed  In  the  four  opening  addresses.  These  were  followed  by  four 
concurrent  workshops  constituting  Session  I.  The  material  developed  In 
the  Session  I workshops  was  summarized  and  presented  to  the  assembled 
Conference.  Session  II  was  composed  of  six  workshops  and  these 
discussions  were  sunmarlzed  and  presented  to  the  assembled  Conference. 

The  objective  of  Session  I was  to  determine  the  critical  problems 


Figure  2: USER’S  PERSPECTIVE 
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and  a ranking  of  Importance  of  these  problems,  as  perceived  by  Conference 
participants  from  their  role  as  providers  of  scientific  and  technical 
Information  or  Intermediaries  In  the  STT  process.  Participants  In  each 
of  the  workshops  addressed  the  theme  of  the  Conference:  "Are  the  providers 
of  scientific  and  technical  Information  adequately  meeting  the  needs  of 
the  user?*  In  doing  so,  the  workshop  served  the  dual  role  of  responding 
to  problems  In  accessing  scientific  and  technical  Information  as 
Identified  by  users;,  and.  Identifying  problems  encountered  by  the  ' 
providers  of  scientific  and  technical  Information.  The  Presiders 
guided  the  discussion  toward  a rank  ordered  list  of  seven  most  critical 

t 

problems  Identified  by  workshop  participants  and  presented  a sunnary  of 
the  discussion  to  the  assembled  Conference. 

The  objective  of  Session  IT  was  to  .discuss  the  elements  of  each 
Issue  ares  raised  during  the  workshops  of  Session  I.  The  presider  served 
the  same  role  as  In  Session  r.  The  Issue  areas  which  emerged  from  the 
problae  analysis  of  Session  F were? 

T.  Economics  - costs,  financing,  cost  barriers,  budgets,  mechanisms 
to  eqauHze  access,  pu611c/pr1vate‘  good,  new  costing  methods,  basis 
research  on  the  use  of  Information. 

Z.  Institutional  Roles  and  Responsibilities  > centralization,  de> 
centralization,  role  definition,  responsibilities,  networking.  Inter- 
national and  Inter-lnstitutlon&l  cooperation,  control,  regulation, 
catalyst. 

3.  Information  about  Resources  Available  • directory,  central 
switching,  selling  skills,  "glut". 
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4.  Availability  - technical  reports,  unpublished  information,  access  to 
governmental  data  bases,  all  non-classified  data,  document  delivery, 
international  document  availability  and  source  document  provision. 

5.  Standards  - common  command  languages,  structures,  service  (document 
delivery  back-up),  software,  formats. 

6:  Delivery  for  Different  Groups  - sophisticated/lay,  foreign  to  English^ 
inter-disciplinary,  scientists  and  decisionmakers,  link  to  public 
policy-making-. 

7.  Education  - citizens,  data  inputters,  retrievers,  decision-makers, 
recognition  of  information  costs,  selling  skills. 

The  findings  of  the  Providers'  Perspective  Conference  are  summarized  as 
Figure  3. 

The  discussions  at  the  Providers'  Perspective  Conference  showed 
different-  emphases  from  those  displayed  at  the  Users'  Perspective 
Conference.  However,  botfr  perspectives  displayed  problems  in  using  and 
providing  STT  in  an  environment  which  supports  the  traditional  STI  system 
as  shown  in  Figure  4.  In  the  traditional  system,  the  scientist  is  the 
generator  and  user  of  disciplinary  scientific  information  for  the 
identification  of  research  opportunities  and  the  validation  of  research. 

The  emerging  perspective  of  a scientific  and  technical  conmuni cation 
system  was  constructed  to  minimize  the  identified  problems,  issues  and 
incorporate  recommendations  of  the  users  and  providers  of  STI.  This 
Is  shown  in  Figure  5. 

The  emerging  model  of  scientific  and  technical  communication  has 
two  objectives  - to  advance  scientific  and  technical  knowledge  and  utilize 
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Objective:  To  advance  scientific  knowledge. 
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scientific  and  technical  knowledge  for  the  advancement  of  society 
by  providing  choices  among  alternative  futures.  On  the  input  portion 
of  the  emerging  model,  scientists  and  interdisciplinary  scientists 
working  with  information  snecialists  organize  their  research  findings  in 
a hierarchy  which  includes  discipline  oriented  information,  problem-oriented 
information  and  integrated  information  for  decision-making  and  policy- 
making. The  hierarchy  of  information  may  be  cross-referenced  and 
synthesized  with  national  economic,  social,  and  cultural  information 
and  international  scientific  and  technical  information.  On  the  output 
side  of  the  emerging  model,  "needers"  of  information  are  directed  to 
the  appropriate  source  of  bibliograohic  data  or  copies  of  a reference. 
"Needers"  conmunicate  with  the  stored  STI  using  standardized  languages, 
a variety  of  communication  devices  and  receive  responses  in  the  media  of 
their  choice..  Information  furnished  through  the  data  base  would  be 
complete  and  validated. 

Inherent  in  a hypothetical  model  such  as  this  are  key  policy  issues 
concerning  economics,  public  and  private  institutional  roles  and 
responsibilities,  the  performance  of  new  functional  activities  - for  example 
the  establishment  of  a central  switching  system,  and  centralized  planning 
and  control . 

Policymakers'  Perspective  Conference 

On  June  2,  1977,  168  participants  at  the  Policymakers'^  perspective 
Conference  addressed  the  question:  "Are  there  current  needs  for  public 
policies  affecting  scientific  and  technical  communication  in  the  United 
States  and  if  so,  what  are  the  options?"  The  question  was  addressed 
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via  four  workshops,  each  oriented  toward  an  issue  area  defined  by  the 
previous  two  conferences^  Ihe  issue  areas  were: 

T.  Whether  there  is  a.  need  for  economic  mechanisms  to  equalize  access 
to  scientific  and  technical  coranunication,  to  decrease  costs,  and  to 
support  research  and  development  on  the  use  and  benefits  of  such  infor- 
matiorr. 

2.  Whether  insitutional  roles  and  responsibilities  should  be  controlled 
by  regulations  - in  e decentralized  or  centralized  manner  - in  networking, 
inter-institutional  cooperation  and  international  arrangement,  including 
the  appropriate  rcTe  for  the  private  sector  and  intergovernmental 
scfence  and  technology  sharing-. 

Whether  the  development  and  support  of  new  functional  activities  for 
scientific  and  technicaT  communication  such  as  validation,  standardization, 
analysis,  synthesfs,  citizen  education  and  a directory  service  or  central 
swftching:  systenr  ta  information  should  be  institutionalized  and  in- 
which  institution. 

4.  Whether  centralized  planning  is  required  to  manage  the  Nation's 
scientific  and  technical  information  resources  to  meet  the  different 
need  levels  and  delivery  requirements  of  the  receivers  who  may  be 
scientists,  deci si on-makers ^ policy-makers  or  non-specialist  citizens. 

Preceding  the  workshops,  each  issue  was  addressed  in  a prepared 
statement  by  prominent  individuals  in  information  science  and  the  associated 
issue  area.  Each  workshop  had  an  assigned  Presider,  Resource  Person,  and 
Assistant-  The  participants  selected  workshops  in  accordance  with  their 
Interests.  During  the  discussion,  the  Presiders  were  encouraged  to 
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solicit  conments  oriented  toward  specific  issues  within  the  area  and 
specific  public  policy  options  for  each  issue.  A summary  of  the  workshop 
discussions  includes  not  specific  policy  choices  but  significant  questions 
which  need  to  be  addressed  prior  to  policy-making.  A summary  of  the 
four  issue  area  discussions  follow. 

1.  Issue  - Eonomic  Mechanisms 

General  Themes 

-accuracy  of  information  re:  sources  and  users  of  STI 
-units  of  measurement  for  policy  and  valuation 

Cost/Value  Relationships 

-cost  of  improvement  vs  exploitation  of  government  STI 
-cost/vaTue  relationships  in  formal  vs  informal  STI  systems 

Structural  Issues 

-relationship  between  STI  policy  and  technology  policy 
-relative  costs  of  formal  vs  informal  STI  systems 
-clarification  and  impact  assessment  of  role  and  responsibilities 
. _ ..of  public  and  private  sectors 

-is  STI  naturally  monopolistic 
-cost  structure  for  STI  production  and  distribution 

Societal 

-should  government  ensure  that  an  "informed"  and  honest  market 

exi sts 

-is  it  government  responsibility  for  labeling  STI  products 

as  "not  necessarily  certified" 

-how  far  should  the  government  go  in  translating  STI  for 

the  conmoffl  citizen 

-should  government  critically  assess  its  own  STI 

-should  STI  be  viewed  as  a universal  input,  requiring  public 

utility  regulation 

2.  Issue  - Roles  and  Responsibilities 

General  Themes 

-is  this  a market  area  or  resource  regulation  area 
-identify  focal  points  for  national  information  policy 
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Procedural 


•standards  and  enforcement 
-International  coordination 
-tax  policies 
-validation 

-need  for  intermediary  Information  gnerator  - synthesis 
-for  whonr  should  Information  be-  provided 

Institutional 

-federal /state  roTes^ 

-different  Institutional  roles 

-sensltlvl^  to  constitutional  .morale  political  restraints. 

-the  press 

-competition  between  private  and  public 

[lemand  Orientation- 

-STT  for  different  publ  Ics 
-state  and  local  ne^s 
-policymaker  needs 
-capacity  to  use 

-new  functions  - role  of  essayist^  book/article  review  and  validation 
-purpose  of  information 
-different  kinds  of  STT 

3-  Issue  - New  Functions 

General  Themes 

-lack  of  attention.  Interest  and  focus  on  Information*  Information 
polity  Is  ad. hoc*  Meed  a set  of  assessment  and  value  criterlia 
for  Information.  Policy  should,  address  access^-announcement-^  ’ . 
Investment,  documentation.  What  are  driving  forces  In  STI? 

-public  relations  activities  required  - demonstrations  effort 
• on  conmunlcatlons  technology,  national  effort  dealing  with 
education  curricula* 

Issues 

-develop  numeric  data  for  recording  locations,  promotion  of  data  centers 
^coordinate  systems,  data  bases  and  networks 
•comnunl cation  within  Information  science  field  - promote,  examine 
and  test  new  approaches  and  demonstrate 
-Improve  education  re:  Information  access  for  all  groups  and 

levels  of  users 

-roles  and  technologies  In  providing  Information  use  dynamic  > 
there  1sn'*’t  a model  of  a future  "system"  to  serve  as  a guide 
for  the  present 
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-STI  transfer  might  be  resolvable  through  information  science 
and  technology  - requires  research  and  communication  between 
scientist  and  information  specialist. 

-quality  of  STI  - ascertained-  through  screening  and  filtering 

devices 

4.  Issue  - Management  of  STI 
General  Themes 

-information  is  a resource  requiring  appropriate  management, 
not  ad  hoc,  fragmentary,  sporadic 
-government  should  exert  leadership  in  establishing  a model 

system  with  a single  classification,  handling,  processing  and 
access  methodology  to  serve  disciplinary,  interdisciplinary 
and  policy  research 

Issues 

-manage  information  as  a resource  as  we  manage  human,  financial 

and  other  resources 

-reduce  barriers  to  sharing  information  underlyine  public  policies, 

science  and  technology 

-improve  federal  government  information  management  and  coordination 

and  standardization  processes 
-orient  STT  systems  to  user  demand 

-perform  R & D to  improve  STI  interface  between-  sources  (producers) 
- - and  users 

-develop  explicit  federal  policy  on  import/ export  of  STI 
-improve  foresight  and  planning  for  crisis  management  through 

STT  improvement 

-develop  management  techniques  to  expand  the  narrow  parochial 
channels  which  restrict  the  flow  of  STI  and  its  utilization 
-disseminate  products  of  federal  R i D expeditiously 
-finance  communication  of  STI  adequately  as- independent  function 

Conferences'  Analysis  and  E valuation 

The  activities  of  the  Forum  at  this  time  involve  synthesis  and 

analysis  of  the  problems  and  issues  described  in  this  report  and  the 

approximately  500  questionaires  collected  during  the  three  conferences. 
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The  Forum  Report,  which  will  be  completed  September  30,  1977  will  include 


a hierarchy  of  problems,  issues  requiring  more  research  and  some  specific 
policy  recommendations. 
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The  following  are  some  general  observations  based  on  the  conferences 
and  associated  activities. 

1.  There  is  wide  Interest  irr  scientific  and  technical  coitmuni cation 
as  shown  by  the  conference  registration  figures,  travel  involved  in 
attendina  conferencesr  a one  day  investnent  of  time,  active  workshop 
discussions  and.  deta.iled  answers  to  the  questional res. 

Z,  Th«*eare  needs  for  public  policy  in  scientific  and  technical 
comnunication  as  expressed  at  all  the  conferences.  In  particularr  the 
Users'  Perspective  Conference  was  characterized  by  appreciation  and 
enthusiasm  for  "thdir'*  -opportunity  to  express  their  needs. 

S.  Public  policy  reconmendations  were  difficult  to  specify.  In  all 
discussions^  there  appeared  to  be  a sense  of  "the  whole  is  bigger  than- 
tbe  parts."  and  none  of  "us"  see  the  whole.  Unwanted  byproduct  effects 
of  premature  policy  reconmendations  introduced  a reluctance  to  state 
reconmendations. 

* 4.  There  appeared  to  be  much  support  for  the  view  that  scientific  and 
technical  comnunication  is  an  entity  requiring  management,,  planning 
and  a future  orientation  as  opposed  to  being  simply  a by-product  of 
another  managed  process  CR  S D funding). 

5.  Scientific  and  technical  comnunication  has  become  more  closely 
related  to  other  information  disciplines  and  "national  information 
policy"  considerations  then  it  was  previously. 

6.  That  the  federal  government  should  exert  greater  leadership  in 
coordinating  its  own  scientific  and  technical  information  systems, 
developing  model  systems  and  interfacing  its  activities  with  the 
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private  sector. 

7.  Recognition  that  scientific  and  technical  conmunication  is  not  now 
a politically  valuable  issue  area. 

8.  Interest  in  greater  sharing  of  STI  among  federal,  state  and  local 
government  jurisdictions  may  provide  greater  political  interest  and  thus 
momentum  to  improvements  in  the  nation's  scientific  and  technical  communi- 
cation capabilities. 

9-.  There  are  many  "players"  with  vested  interest  in  scientific  and 
technical  conmuni cations,  whose  contributions,  roles  and  reactions  must 
be  appreciated  and  anticipated  in  policy-making  considerations. 

Forunr  Approach 

The  Forunr  approach  to  software  development  is  advantageous  because 
it  permits  one  to  go  to  the  "grass  roots"  level  for  data  collection. 

In  addition,  when  the  federal  government  is  involved^  it  is  valuable  to 
study  any  existing  system,,  as  it  is  perceived  by  citizens  outside  the 
federal  bureaucracy.  And,  as  their  attention  and  views  are  expressed, 
their  interest  and  support  to  the  project  may  be  won.  Problem  definition 
and  criticality  are  perceived  differently  by  different  “players"  in  the 
hierarchy  of  any  system.  It  is  useful  to  analyze  these  different 
perceptions  to  clarify  significant  problems,  as  measured  by  established 
criteria.  The  conference  format  permits  or^to  informally  assess  the 
different  ''players"  in  the  process  and  define  the  power  structure. 

There  are  some  disadvantages  to  a Forum  approach,  some  of  which 
may  be  overcome.  Firstly,  it  is  difficult  tobe  certain  that  the  conference 
is  attended  by  a representative  sample  of  the  population  one  is 
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interested  in.  This  is  particularly  true  when  conferences  are  supported 

• I 

by  federal  monay  and  an  open  door  policy  nust  prevail.  At  such  meetings, 
specialized  groups  may  overpower  discussion?  and  distort  the  findings. 
Full  attention  to  conference  activities  and  inattention  to  the  operations 
of  large  institutions  that  are  involved  may  preclude  the  detection  of 
"behind  the  scenes.*  activities.  These  may  neutralize  or  sabotage  the 
outcomes  of  the  conference. 

Finally,  there  are  certain  ingredients  which  contribute  to  a 
Forum's  success.  These  are: 

1.  Identification  of  knowledgable  participants. 

2.  Appropriate  timing-  for  conferences  and  Forum  recomnendations. 

Tl  ConRitment  of  an  organization  to  the  Forum  study. 

MilTingness  to  accept  findings  in  return  for  participants'  time 

investment. 

S.  rcreening*  devices  of  inpu't  from  vested  interests. 

S.  Objective  environment  and  conduct  of  the  Forum  activities. 

7.  Support  of  the  power  structure. 
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VII 


SOFTWARE  LIFE  CYCLE  MANAGEMENT  WORKSHOP 
August  21-23,  1977 

Confereuce  Coordinator:  Dr.  Bryce  Elkins 
Workshop  Directors:  Prof.  M.  M.  Lehman 

Col.  L.  A.  Putnam 


5:00  p.  m. 


Social  Hour 


6:00  p.m. 


Dinner 


7:00  p.m. 


Staff  meeting 


August  22,  1977 


7:30  a.  m. 


Breakfast 


8:00  a.  m. 


General  Session 


9:15  a.  m. 


Technical  Sessions 


10:45  a.m. 


Coffee  Break 


11:00  a.  m. 


General  Session 


12:00  noon 


Lunch 


1:00  p.m. 


3:00  p.  m. 
3:15  p.m. 
5:00  p.  m. 


6:00  p.m. 


Technical  Sessions 


Coffee  Break 


General  Session 


Social  Hour 


Dinner 


7:30  p.m. 


General  Session 


After  Dinner  Address,  Dr.  Staudhammer 
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Technical  Sessions 
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Plenary  Session 
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